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I .  INTRODUCTION 

A.  BACKGROUND 

"The  Puny  Payoff  from  Office  Computers."  That  was  the 
cover  story  for  Fortune  Magazine  26  May  1986.  Why  is  this?  The 
computer  age  was  to  bring  increases  in  white-collar 
productivity  as  earlier  generations  of  technological 
innovation  did  for  other  sectors,  e.g.  farming  and 
manufacturing.  So  far,  business  investment  of  hundreds  of 
billions  of  dollars  in  computer  systems  has  failed  to 
significantly  improve  productivity.  There  has  been  little  or 
no  change  in  white-collar  productivity  since  the  sixties, 
where  85-90  percent  of  information  technology  is  invested 
(Fortune  26  May  1986)  .  Combined  with  the  fact  that 
"information  technology  is  now  the  largest  line  item  in 
Corporate  America' s  capital  spending  budget  .  .  .  Companies  need 
to  develop  better  ways  of  measuring  office  productivity  and 
the  contribution  that  managers  and  other  white-collar  workers 
make  to  the  bottom  line."  (Business   Week,    26  November  1990) 

B.  OBJECTIVES 

Evaluation  of  information  systems  is  necessary  to 
effectively  manage  them  as  organizational  resources.  To  do 
this  an  organization  must  be  able  to  measure  the  costs  and 
benefits  related  to  its  systems.  This  allows  decisions  on 


design  and  implementation  to  "be  made  in  the  same  way  as  every 
other  firm  investment  decision,  i.e.  based  on  its  contribution 
to  the  success  of  the  firm."  (Marsden  and  Pingrey,  1988) 

Various  evaluation  methods  have  been  used  to  assess 
information  system  effectiveness.  The  one  of  special  interest 
for  this  thesis  is  user  satisfaction.  A  number  of  articles  in 
IS  literature  cite  user  satisfaction  as  a  surrogate  for 
effectiveness  (Ives,  Olson  and  Baroudi,  1983;  Pearson,  1977; 
Seward,  1975;  Hamilton  and  Chervany,  1981).  The  desire  for  a 
surrogate  stems  from  the  fact  that  unlike  transaction 
processing  systems,  data  on  IS  use  provides  little  information 
on  the  effectiveness  of  the  white  collar  world.  The  issue  here 
is  whether  opinions  about  a  system  such  as  user  satisfaction, 
are  the  same  thing  as  effectiveness  of  a  system.  MIS  is  a 
multidisciplinary  field  spanning  technical  sciences, 
management  science,  psychology  and  other  social  sciences. 
Problems  can  arise  when  techniques,  theories  or  results  from 
a  study  in  one  field  are  used  in  another.  Management  science 
has  had  difficulty  in  establishing  a  link  between  satisfaction 
and  productivity.  This  thesis  will  review  the  MIS  literature 
that  attempts  to  show  a  relationship  between  user  satisfaction 
and  system  effectiveness.  It  will  examine  the  development  of 
this  relationship  in  the  MIS  literature.  Specifically,  it  will 
seek  answers  to  the  following  questions: 


Is  user  satisfaction  a  useful  measurement  of  IS 
effectiveness? 


2.  What  are  the  various  meanings  of  user  satisfaction 
in  IS  literature? 

3.  Does  the  IS  literature  now  assume  that  measuring 
user  satisfaction  is  equivalent  to  measuring 
effectiveness  for  IS,  if  so,  does  it  reference 
previous  literature? 

4.  Are  other  methods  used  to  measure  effectiveness? 
What  are  they?  Does  the  author  give  a  reason  for 
the  choice? 


C .   SCOPE 

This  thesis  will  investigate  the  measurement  :  f 
information  system  effectiveness  over  the  past  20  years.  It 
examines  what  methods  have  been  employed  in  the  MIS 
literature.  In  particular,  attention  is  focused  on  the  use  of 
user  satisfaction  as  a  surrogate  for  effectiveness  by  the  MIS 
community.  Specific  areas  of  interest  include: 


1.  What  method  of  measuring  system  effectiveness  was 
employed? 

2.  How    user     satisfaction    was     initially 
determined/def ined. 

3.  The    evolution    of    the    user    satisfaction- 
effectiveness  relationship  and  its  current  state. 

4.  How  and  why  user  satisfaction  is  employed  as  a 
surrogate  for  effectiveness. 

5.  Whether  the  employment  of  user  satisfaction  was 
justified,  i.e.,  correct. 


The  data  gathered  will  provide  information  as  to  the 
various  methods  of  measuring  IS  effectiveness  as  well  as 


information   regarding   the   recent   prevalence   of   user 
satisfaction  as  the  preferred  method. 

D .   METHODOLOGY 

A  literature  review  will  be  undertaken  to  document  the 
development  of  the  user  satisfaction-effectiveness 
relationship  and  the  use  of  user  satisfaction  as  a  surrogate 
for  IS  effectiveness.  It  attempts  to  trace  the  origin  of  the 
relationship  through  to  its  current  state.  Attention  will  be 
focused  on  these  questions: 

1.  How  was  effectiveness  measured? 

2.  Were  other  methods  considered  and  if  so  were  there 
references  cited? 

3.  If  user  satisfaction  was  chosen: 


a.  Was  the  purpose  to  show  user  satisfaction  as 
equivalent  to  effectiveness?  or, 

b.  Did  the  study  assume  equivalence  and  if  so  did 
it  provide  a  reference? 


Defining  and  classifying  what  user  satisfaction  is  and  is 
not  and  what  it  can  and  cannot  be  used  for  was  accomplished  by 
drawing  analogies  from  the  sociological  research  by  Denzin 
(1989)  and  Phillips  (1971)  focusing  in  particular  on  the 
descriptions  and  limitations  of  the  survey  technique.  This 
parallel  was  felt  justified  based  on  the  common  psychological 
nature  of  the  measure  and  the  technique  used  for  its 
employment,  the  survey,  between  sociological  research  and  the 


user  satisfaction  measure  of  IS  research.  Basically,  it  was 
determined  that  user  satisfaction  as  an  opinion,  i.e., 
subjective  measure  should  not  be  employed  as  an  equivalent  to 
effectiveness,  an  action. 

E.   BENEFITS 

User  satisfaction  has  aggressively  been  developed  as  the 
primary  method  of  measuring  IS  effectiveness.  MIS  needs  to 
know  if  this  is  a  proper  measurement  of  its  work.  What  we  know 
about  information  systems  we  know  through  the  methods  by  which 
we  observe  them  and  the  concepts  by  which  we  mentally  model 
them.  If  the  concepts  are  erroneous  we  are  vulnerable  to 
investing  in  systems  that,  in  reality,  are  ineffective. 


II.  BACKGROUND 

A.   WHY  MEASURE  IS  EFFECTIVENESS? 

"Information  technology  is  now  the  largest  line  item  in 
Corporate  America's  capital  spending  budget,  accounting  for 
nearly  a  third  of  current-dollar  equipment  outlays."  (Business 
Week,  2  6  November  1990)  The  need  to  understand  and  measure 
information  system  effectiveness  results  from  the  fact  that 
"U.S.  business  has  spent  hundreds  of  billions  of  dollars  on 
them  [computers],  but  white-collar  productivity  is  no  higher 
than  it  was  in  the  late  sixties."  (Fortune,    26  May  1986) 

Organizations  have  invested  in  information  technology  to 
increase  productivity,  specifically  white-collar  productivity. 
"Economists  calculate  that  about  85%  of  investment  in 
information  technology  is  made  by  the  service  [white-collar] 
sector.  Yet  non-manufacturing  productivity  has  risen 
cumulatively  just  1.9%  over  the  past  10  years.  ... 
AUnf ortunately, '  observes  Morgan  Stanley  economist  Stephen 
Roach,  ^the  service  sector  has  devoted  far  more  effort  to 
deploying  computers  than  in  determining  the  precise  ways  they 
can  be  used  to  enhance  white-collar  productivity' 
Companies  need  to  develop  better  ways  of  measuring  office 
productivity  and  the  contributions  that  managers  and  other 


white-collar  workers  make  to  the  bottom  line."  (Business  Week, 
26  November  1990) 

Clearly  then,  increasing  the  effectiveness  of  information 
systems  is  a  primary  objective  of  business  organizations.  To 
do  so,  they  need  to  know  what  factors  affect  and  contribute  to 
system  effectiveness.  It  is  this  understanding  of 
effectiveness  which  MIS  research  must  provide  so  that 
increases  in  productivity  and  thus  profits  are  realized, 
thereby  justifying  the  current  magnitude  of  investment  in 
information  systems  by  U.S.  businesses. 

1.  The  Goal  of  MIS  Research 

The  goal  of  research  is  to  increase  the  understanding 
of  some  relationship  or  situation.  The  goal  of  MIS  research  is 
to  better  describe  and  understand  the  factors  and 
contributions  of  information  systems  to  an  organization.  "MIS 
research  is  the  systematic  investigation  of  the  development, 
operation,  use  and/or  impact  of  an  information  (sub) system  in 
an  organizational  environment."  (Ives,  Hamilton  and  Davis, 
1980)  .  The  goal  of  information  systems  is  to  increase  the 
effectiveness  of  the  organization,  where  effectiveness  is  the 
ability  of  performance  to  meet  the  goals  and  objectives  set  by 
the  organization.  The  purpose  of  evaluation  is  to  assess  or 
appraise  the  value  of  an  activity  and  measure  its 
effectiveness  (Arnovick  and  Gee,  1978). 


How  MIS  measures  effectiveness  is  critical  to  the 
evaluation  process  and  ultimately  the  overall  understanding  of 
information  systems. 

2 .  Measuring  Effectiveness 

Hamilton  and  Chervany  (1981)  present  two  different 
views  of  system  effectiveness.  Each  has  its  own  meaning  of 
system  effectiveness  and  how  it  should  be  measured.  In  the 
goal-centered  view  "Effectiveness  is  determined  by  comparing 
performance  to  objectives."  An  example  of  this  approach  would 
be  cost-benefit  analysis.  The  second  is  the  system-resource 
view  where  effectiveness  is  seen  as  "attainment  of  a  normative 
state,  e.g.,  standards  for  *good'  practices"  rather  than 
establishing  specific  task  objectives. 

The  complementary  nature  of  these  views  is  similar  to 
that  of  summative  and  formative  evaluation  approaches.  The 
focus  of  summative  evaluation  is  the  accomplishment  of 
objectives.  These  are  tangible  factors  capable  of  objective 
measurement.  Formative  evaluation  is  assessment  of  quality  and 
consist  of  intangible  factors  measured  subjectively.  "Both 
formative  and  summative  evaluation  approaches  are  typically 
used  in  providing  evaluative  information  on  system 
effectiveness."  (Hamilton  and  Chervany,  1981). 

It  is  important  to  point  out  the  significance  of  the 
dual  nature  of  the  evaluation  process  and  the  contributions 
and  limitations  of  each  approach. 


a.  Goal-Centered  View 

Basically  the  goal-centered  view  of  system 
effectiveness  focuses  on  the  efficiency  of  the  system.  It 
assesses  system  performance  with  respect  to  established  system 
and/or  organizational  objectives.  Quantifiable  (tangible) 
benefits  provide  objective  measures  of  performance.  Like  the 
summative  evaluation  approach,  the  goal-centered  view  provides 
"information  on  the  systems  outcomes,  or  ends,  to  support 
decisions  to  continue,  adopt  or  terminate  the  system." 
(Hamilton  and  Chervany,  1981) .  That  is,  the  objectives  set  by 
the  organization  are  the  critical  factors  against  which  the 
system  success  or  failure  is  judged. 

b.  System-Resource   View 

The  system-resource  view  uses  subjective  measures 
of  intangible  benefits  to  assess  the  effectiveness  of  the 
system.  Like  the  formative  evaluation  approach,  the  system- 
resource  view  provides  supplementary/explanatory  information 
about  the  system  (Hamilton  and  Chervany,  1981) . 

B.   THE  REASON  FOR  THIS  STUDY 

Hamilton  and  Chervany' s  (1981)  dual  view  of  system 
effectiveness  identifies  these  two  system  evaluation 
approaches  as  separate  yet  complimentary.  This  complimentary 
relationship  is  analogous  to  the  one  between  summative  and 
formative  evaluation  approaches  where  information  provided  by 
the  formative  evaluation  helps  "improve  the  means  and  process 


to  accomplish  objectives  and  interpretation  of  summative 
evaluation  results."  They  further  explain  (1981)  that 
objective  measures  are  provided  by  the  "formal  evaluation 
approaches"  while  "perceptions  of  system  effectiveness"  are 
gained  by  the  informal  ones  which  are  "necessary  and  helpful 
for  calibrating  the  credibility  of  information  on  MIS 
evaluation . " 

Assessing  the  accomplishment  of  official  goals  and 
objectives  and  thus  the  overall  effectiveness,  or  success,  of 
the  system  is  the  function  of  the  formal  evaluation 
approaches,  i.e.,  goal-centered  view,  summative  evaluation, 
and  objective  measures.  The  informal  approaches;  the  system- 
resource  view,  formative  evaluation,  and  subjective  measures 
provide  supplementary  information,  recognizing  that  "systems 
have  other  functions  and  have  other  consequences  besides 
accomplishment  of  official  objectives"  (Hamilton  and  Chervany, 
1981) . 

The  message  here  is  that  the  system-resource  view, 
formative  evaluation  and/or  subjective  measures  are  informal 
approaches  and  are  at  best  supplementary  measures  to  "explain" 
or  "help  ...  and  aid."  The  point  being  that  the  goal-centered 
view,  summative  evaluation  and  objective  measures  are  formal 
evaluation  approaches  and  are  the  primary  means  of  determining 
the  success  or  failure  of  the  system,  to  be  supplemented  by 
their  "informal"  counterparts. 
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Relevant  to  the  current  study  is  the  subjective  measure, 
user  satisfaction,  which  is  a  perception  of  system 
effectiveness.  Although  Hamilton  and  Chervany  suggest  that 
"The  user  attitude  survey  [user  satisfaction]  is  a  formal 
approach  to  quantify  subjective  perceptions  of  system 
effectiveness."  this  thesis  contends  that  user  satisfaction  is 
none  the  less  just  a  supplementary  measure  of  system 
effectiveness . 

The  hazard  in  employing  user  satisfaction  as  a  prime  means 
of  evaluating  the  effectiveness  of  information  systems,  to  the 
exclusion  of  assessing  costs  and  benefits,  is  that  it  is  an 
opinion-based  technique.  An  example  of  the  hazard  is 
illustrated  by  international  surveys  of  students'  opinions  of 
their  math  competency.  In  such  surveys,  U.S.  high  school 
students  rate  themselves  among  the  worlds  best.  In  actual 
testing  of  math  skills,  U.S.  high  school  students  are  among 
the  world's  worst.  The  opinion  is  not  the  same  as  the  skill. 
Similarly,  users  of  information  systems  may  have  numerous 
motives  to  claim  satisfaction  with  these  systems.  Yet,  these 
opinion  reports  are  not,  conceptually,  the  same  thing  as  an 
assessment  of  a  system's  productivity  in  terms  of  inputs  and 
outputs  or  its  quality  in  terms  of  better  management 
decisions.  Indeed,  in  two  studies  where  both  effectiveness  and 
satisfaction  were  measured,  the  findings  were  contradictory, 
Gallupe  and  DeSanctis  (1988),  and  Aldag  and  Power  (1986).  It 
is  the  distinction  between  primary  and  supplementary  measures 
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that  will  serve  as  criteria  for  classifying  articles  reviewed 
in  this  thesis.  The  purpose  will  be  to  trace  the  genesis  and 
evolution  by  which  user  satisfaction  crowded  out  cost-benefit 
analysis  . 
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III.  METHODOLOGY 

A.   SELECTION  OF  SYSTEM  EFFECTIVENESS  STUDIES 
1 .   Selection  of  Articles  for  Study 

The  concept  of  this  study  has  as  its  impetus, 
questions  raised  in  a  previous  thesis  on  information  system 
effectiveness  by  Mustofa  (1989).  Therefore,  it  seemed 
reasonable  to  begin  article  selection  with  those  identified  as 
dealing  with  systems  effectiveness  in  the  preceding  study.  To 
these  were  added  articles  published  after  1988,  the  last  year 
included  in  the  Mustofa  study.  These  new  articles  were 
collected  by  the  same  means,  which  will  be  briefly  discussed. 
(The  reader  may  refer  to  Chapter  III  Methodology,  Mustofa 
(1989)  for  a  detailed  account  of  the  search  and  selection 
process . ) 

A  consensus  of  journal  names  from  three  studies 
reporting  on  articles  published  in  the  MIS  field  was  selected. 
The  starting  point  was  1970  and  continued  through  1988.  A 
DIALOG  search  from  the  INSPEC  database  was  conducted  by  the 
Dudley  Knox  Library  at  the  Naval  Postgraduate  School.  The 
bibliographic  search  was  on  the  following  key:  (Productivity 
or  Effectiveness)  and  (Measures  or  Measurement  or  Assess  or 
Assessment)  and  Information  System.  This  automated  search 
turned  up  a  limited  number  of  useful  articles,  fewer  than  25. 
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Therefore,  a  manual  search  was  conducted  examining  the 
periodicals  on  the  consensus  list  between  the  years  1970  and 
1988.  Indices  or  every  issue  of  periodicals  without  an  index, 
were  examined  for  articles  possibly  dealing  with  information 
system  effectiveness.  This  search  turned  up  118  articles  as 
well  as  the  reason  for  the  small  number  found  during  the  on- 
line search,  the  limited  number  of  key  word  synonyms  for 
effectiveness  and  productivity.  Additionally,  articles  were 
obtained  from  friends  and  colleagues  aware  of  the  situation  or 
involved  in  related  work  as  well  as  those  published  subsequent 
to  1988  up  to  July  1990. 

2.  Methods  of  Evaluating  System  Effectiveness 
a.  Knowledge 

Knowledge  accumulation  is  an  important  aspect  in 
"the  relationship  between  actions  and  outcomes"  (Goldstein  et 
al.,  1990).  Various  methods  of  knowledge  accumulation  have 
been  proposed  and  summarized  by  Goldstein  et  al.  (1990)  :  Weick 

(1979)  believes  that  "individuals  make  sense  of  events 
retrospectively, "  that  is,  they  assimilate  information  about 
the  event  (s)  and  then  explain  them  retrospectively  to 
understand  them.  Salancik  and  Porac  (1986)  have  identified  a 
three  step  process;  gather  data  to  understand  cause  and  effect 
relationships,  build  a  knowledge  base  of  these  relationships, 
and  use  them  to  come  to  generalized  statements.  Isenberg 

(1986)  identified  "a  cycle  of  reasoning  and  action"  using 
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existing  knowledge  identify  any  new  information  required  and 
then  act  and  update  through  feedback.  Lord  and  Foti  (1986) 
describe   a   knowledge   representation   scheme   classifying 
different  environmental  elements  which  are  used  as  prototypes 
and  organized  hierarchically  for  decision  making. 

The  amount  of  knowledge  accumulated  is  linked  to 
job  performance.  Experts  were  found  to  have  not  only  more 
information  available  to  them  but  it  was  also  more  useable 
i.e.,  they  had  more  and  better  descripters,  and  more  inter- 
category  information  (Lurigio  and  Carroll,  1985) .  Goldstein  et 
al .  (1990)  used  a  knowledge  classification  scheme  of 
declarative,  procedural  and  causal  knowledge.  "Declarative 
knowledge  consists  of  facts  and  definitional  descriptions  of 
categories  .  .  .  Procedural  knowledge  is  the  heuristics  and 
rules  that  guide  a  person's  actions  ...  Causal  knowledge 
consists  of  the  description  of  the  existence,  strength  and  the 
direction  of  the  relationships  among  categories  and 
strategies . " 

b.     Cost-Benefit-Analysis 

"A  cost-benefit-analysis  (CBA)  is  essentially  a 
comparison  between  two  states.  Proposed  new  system  costs  and 
benefits  are  usually  compared  with  those  of  current  systems 
whether  they  be  manual,  partly  computerized  or  fully 
computerized."  (Lincoln,  1986).  There  is  usually  little 
difficulty  identifying  costs  of  the  systems  as  these  are 


15 


usually  well  documented.  However,  identification  of  benefits 
and  subsequently  assigning  a  value  to  them  has  proven  to  be 
difficult.  Basically  there  are  two  types  of  benefits,  tangible 
and  intangible.  Tangible  benefits  are  relatively  easy  to 
quantify  and  assign  a  value.  Intangibles,  on  the  other  hand, 
have  proven  to  be  hard  to  quantify  and  when  values  were 
assigned  they  have  been  viewed  skeptically.  Often  they  are  not 
even  considered  in  the  problem.  There  are  different 
philosophies  to  the  valuation  of  intangible  benefits.  Matlin 

(1979)  argues  for  "placing  dollar  amounts  on  all  intangible 
benefits"  stating  that  "Difficulty  in  finding  a  value  is  not 
an  appropriate  reason  for  avoiding  the  subject.  It  is  vitally 
important  to  derive  techniques  for  valuing  benefits  of 
information  system  investments.  Even  an  approximation  of  the 
values  will  provide  managers  an  opportunity  to  make  informed 
decisions  about  future  investments.  Without  these  values 
decisions   may  be   made   without   appropriate   information." 

(Matlin,  1979) 

Another  approach  is  made  by  Hall  and  Lincoln 

(1976).  They  consider  only  quantifiable  benefits,  estimate 
them  conservatively  and  check  "the  financial  results  for 
accuracy  and  plausibility"  reasoning  that  senior  management 
has  been  made  skeptical  of  CBA  by  "many  examples  of  quick, 
glib,  cost  justification  exercises." 

The  value  obtained  is  then  used  in  a  comparison 
between  pre  and  post  system  introduction,  but  requires  a 
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stable  environment  or  a  control  group  in  place  to  be  a  valid 
measure.  Alternatively,  the  value  may  be  compared  against 
another  system,  which  however,  requires  "a  reconstruction  of 
the  state  of  affairs  assuming  either  an  alternative  system  is 
in  place  or  the  activity  was  not  undertaken."  (Lincoln,  1976) . 
Determining  the  basis  for  comparison  i.e.,  previous  system, 
alternative  technology  or  manual  system  and  the  level  of 
service  is  complicated,  difficult  and  uncertain.  Senior 
executives  often  require  initial  comparison  to  a  manual 
system.  Its  advantages  include  zero-based  comparison  and 
familiarity  due  to  a  large  database  of  similar  applications 
over  many  years . 

CBA  has  been  criticized  because  of  the  difficulty 
and  subjectivity  in  identifying  and/or  evaluating  the 
intangible  benefits  of  unstructured,  ad  hoc  decisions  and 
obtaining  determinable  data  supporting  system  success  but 
unrecorded  by  an  organization  (Ives  et  al.,  1983) . 
c.     Value  Analysis 

Value  analysis  is  similar  to  user  satisfaction  upon 
which  it  is  based  but  is  preferred  in  evaluating  DSS  (Money  et 
al.,  1988)  .  Value  analysis  is  employed  in  evaluating  DSS  and 
begins  with  the  premise  that  "information  is  data  of  value  in 
decision  making"  (Yovits  and  Foulk,  1985).  They  also  suggest 
"that  information  gives  rise  to  observable  effects  through  the 
decision  making  process  and  information  feedback  exists  so 
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that  the  decision  maker  (DM)  will  adjust  his  assessment  of  the 
decision  situation  for  later  decisions"  (Yovits  and  Foulk, 
1985)  .  Money  et  al .  (1988)  using  the  method  of  conjoint 
measurement  convert  subjective  judgement  of  the  value  of 
intangible  benefits  into  scores  for  analysis. 

d.     Profit  Maximization 

A  key  objective  of  an  organization  is  profit 
maximization.  Information  systems  design  should  focus  on 
achieving  that  goal.  Marsden  and  Pingrey  (1988)  argue  that 
decisions  regarding  IS  be  made  as  any  other  investment 
decision,  "its  contribution  to  the  success  of  the  firm."  They 
develop  a  goal  drive  mathematical  process  model  which  seeks 
the  goal  of  profit  maximization.  Using  values  of  revenue 
enhancement  or  cost  reduction  of  various  information  products 
and  costs  of  the  necessary  man-machine  inputs  of  those 
products,  they  maximize  the  difference.  Citing  numerous 
references  in  the  economics  and  IS  fields  to  support  their 
technique,  Cron  and  Sobol  (1983)  use  multiple  documented 
financial  performance  measures;  pretax  return  on  assets, 
return  on  net  worth,  pretax  profits  as  a  percent  of  sales  and 
average  five-year  growth  as  indicators  of  performance,  i.e., 
effectiveness . 

e       Use 

Use  of  a  computer  system  is  expected  to  improve 
performance   of   its   users   (Lucas,   1974)  .   Therefore   the 
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relationship  between  use  of  a  system  and  performance 
(effectiveness)  is  investigated.  Effective  systems  are  assumed 
to  attain  higher  use  than  less  effective  ones.  However,  this 
requires  that  use  of  the  system  is  optional  as  for  a  DSS.  With 
mandatory  use  of  the  system  the  relationship  between  use  and 
effectiveness  fails.  Criticism  of  use  as  a  measurement  for 
effectiveness  comes  from  the  fact  that  it  measures  only  the 
input  to  the  system,  where  as  the  desired  value 
(performance/effectiveness)  concerns  the  output  of  the  system. 

f.  Productivity 

Productivity  as  a  measure  of  system  effectiveness 
is  a  before  and  after  comparison  of  some  performance 
objective (s) .  There  are  various  levels  of  sophistication  in 
defining  these  objectives,  from  measuring  simple  letter 
production  using  a  word  processing  system  (Kapsales,  1986)  to 
sophisticated  regression  models  of  multiple  variables 
(Englander  and  Englander,  1985) . 

g.  User  Satisfaction 

User  satisfaction  is  a  measure  of  the  users 
subjective  evaluation  of  factors  concerning  an  information 
system's  utility.  These  factors  originate  from  work  on 
organizational  behavior  by  Cyert  and  March.  They  argue  that 
the  users  "need  for  information"  is  either  reinforced  or 
frustrated  by  the  information  system,  the  success 
(effectiveness)  of  which  is  the  users  sense  of  satisfaction. 
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Most  user  satisfaction  instruments  use  the  semantic 
differential  technique  developed  by  Osgood,  Suci  and 
Tannenbaum,  e.g.,  the  Shult z-Slevin  questionnaire.  However, 
"Most  of  those  that  existed  prior  to  an  empirically  derived 
methodology  developed  by  Pearson  (1977)  have  been  heavily 
criticized"  (Evans  et  al .  ,  1988).  Bailey  and  Pearson  (1983) 
developed  a  tool  from  Pearson's  1977  dissertation,  combining 
factors  from  "a  review  of  22  studies  of  the  computer/user 
interface."  The  Bailey  and  Pearson  tool  is  based  on  Wanous  and 
Lawlers  model  defining  satisfaction  as  "the  sum  of  the  user's 
weighted  reactions  to  a  set  of  factors, 

sj  =  E  Ru  ^ 

where 

Rij  =  The  reaction  to  factor  j  by  individual  i. 

Wij  =  The  importance  of  factor  j  to  individual  i. 
This  model  suggests  that  user  satisfaction  is  the  sum  of  ones 
positive  and  negative  reactions  to  a  set  of  factors."  (Bailey 
and  Pearson,  1983)  .  Bailey  and  Pearson  also  utilize  the 
semantic  differential  technique  of  Osgood  et  al. 

Ives  et  al.  (1983)  reduced  the  Pearson  tool  to  "a 
^short  form'  of  the  instrument  for  use  when  time  is  limited 
and/or  only  an  overall  assessment  of  UIS  [user  information 
satisfaction]  (rather  that  analysis  of  particular  symptoms  of 
a  problem)  is  adequate." 
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h.     Multiple   Criteria. 

The  general  intent  of  multiple  criteria  studies  is 
to  more  thoroughly  "examine  the  significance  of  various 
information  system  characteristics"  (Dickson  et  al.,  1977)  for 
"an  easy-to-use  instrument  for  improving  the  effectiveness  of 
information  systems"  (Evans  et  al.,  1988)  .  Proponents  of  the 
multiple  criteria  approach  conclude  that  "Information  systems 
are  a  product  of  multiple  factors"  (Mansour  and  Watson,  1980)  . 
In  general,  some  combination  of  system  performance,  goal 
attainment  and  user  criteria  is  employed.  The  intent  being  to 
more  fully  describe  and  define  the  IS  environment,  thereby 
more  accurately  evaluating  the  IS  effectiveness  (Mansour  and 
Watson,  1980;  Chandler,  1982;  Evans  et  al . ,  1988).  Chandler 
suggests  information  systems  be  evaluated  from  two 
perspectives,  one  being  the  computer  system  domain  the  other 
the  user  domain,  each  having  its  own  goals  and  measures.  Evans 
et  al.  conclude  that  by  applying  different  methodologies  the 
various  results  "can  work  separately  and  in  harmony  with  each 
other"  (Evans  et  al.,  1988). 

3 .  Evolution  of  the  User  Satisfaction  Tool 

Cyert  and  March  (1963)  are  credited  by  Bailey  and 
Pearson  (1983) ,  and  Ives,  Olson  and  Baroudi  (1983)  with  being 
the  source  of  the  user  information  satisfaction  (UIS)  concept. 
Basically,  they  argue  that  "when  an  organization  discovers  a 
solution  to  a  problem  by  searching  in  a  particular  way,  it 
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will  be  more  likely  to  search  in  that  way  in  future  problems 
of  the  same  type;  when  it  fails  to  find  a  solution  by 
searching  in  a  particular  way,  it  will  be  less  likely  to 
search  in  that  way  in  future  problems  of  the  same  type." 
Drawing  inference  from  this  to  information  systems,  it  is  held 
that  the  ability  of  the  system  to  meet  the  needs  of  the  user 
either  reinforces  or  frustrates  the  users'  sense  of 
satisfaction  with  the  system.  Therefore,  it  is  suggested  that 
"Satisfaction  of  users  with  their  information  systems  is  a 
potentially  measurable,  and  generally  acceptable,  surrogate 
for  utility  in  decision  making"  (Ives,  Olson  and  Baroudi, 
1983)  . 

User  satisfaction  is  a  subjective  measure  of  system 
success.  It  relies  on  the  users  perceived  value  of  the 
information  system's  effectiveness  to  be  a  surrogate  measure 
for  objective  measures  deemed  to  difficult  to  obtain. 

a.     Early  User  Satisfaction  Measures 

The  early  measures  of  user  satisfaction,  those 
through  1977,  had  varied  terminology  for  the  concept  of  user 
satisfaction,  for  example:  "appreciation"  (Swanson,  1974), 
"user  attitudes"  (Lucas,  1974)  ,  "user  perceptions"  (Gallagher, 
1974;  Schewe,  1976),  "perceived  success"  (Edstrom,  1977), 
"user  evaluation"  (Dickson,  Senn  and  Chervany,  1977) . 
Generally  these  measures  were  uniquely  "designed"  for  the 
particular  study  that  employed  it.  Often  the  definition  and 
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construct  of  the  variable  were  not  described  in  the  article, 
see  Appendix  E,  Table  VII. 

b.  The   Shultz-Slevin   Questionnaire 

The  Shultz-Slevin  Questionnaire  is  based  on  the 
Likert  and  semantic  differential  techniques.  The  Likert 
instrument  "consists  of  a  number  of  statements  to  which  the 
respondent  indicates  how  strongly  he  agrees  or  disagrees  on  a 
five-point  or  seven-point  scale"  (Shultz  and  Slevin,  1975) . 
The  semantic  differential  is  a  scale  that  "consists  of  the 
statement  of  a  concept  to  which  an  individual  responds  by 
checking  a  set  of  bipolar  adjectives;  good-bad."  The  Shultz- 
Slevin  Questionnaire  became  the  prevalent  method  in  the 
studies  reviewed  fro^m  1978  to  1983.  The  term  user  satisfaction 
became  more  common,  although  each  study  generally  still  had 
its  own  definition  of  "user  satisfaction."  However,  the 
construct  of  the  measure  began  to  standardize  around  the 
Shultz-Slevin  Questionnaire,  see  Appendix  E,  Table  VII. 

c.  The  Pearson  Methodology  and  the  Bailey  and  Pearson 
Instrument 

Examining  Table  VII  with  respect  to  studies 
published  after  1983  shows  that  either  the  Pearson  Methodology 
or  the  Bailey  and  Pearson  Instrument  was  almost  the  exclusive 
measure  of  user  satisfaction  when  evaluating  information 
system  effectiveness.  The  Bailey  and  Pearson  Instrument  is 
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described   in   section   (2),   Methods   of   Evaluating   System 

Effectiveness  . 

Evans  et  al.  (1988)  sum  up  the  evolution  of  user 

satisfaction  this  way: 

Instruments  to  measure  user  satisfaction  are  scarce.  Most 
of  those  that  existed  prior  to  an  empirically  derived 
methodology  developed  by  Pearson  (Pearson,  1977)  have  been 
heavily  criticized  (Ives,  Olson  and  Baroudi,  1982). 
Pearson's  Methodology,  which  has  received  wide  acceptance, 
uses  a  semantic  differential  technique  to  rate  39  factors 
which  cause  satisfaction  or  dissatisfaction  with 
information  systems.  The  39  factors  were  identified  based 
on  extensive  literature  review  and  expert  opinion.  (Evans 
et  al. ,  1988) . 

4  .  Selecting  Articles  Dealing  With  User  Satisfaction 

The  compilation  of  -articles  for  this  study  obtained 
from  the  search  found  96  as  possibly  dealing  with  user 
satisfaction  or  system  effectiveness.  These  then  were  the 
studies  initially  reviewed  by  this  study  and  are  listed  in 
Appendix  A. 

a.  User  Satisfaction  Studies 

Reading  the  96  articles  in  Appendix  A  yielded  40 
articles  dealing  with  user  satisfaction  or  one  of  its  aliases 
(as  described  in  the  preceding  section)  .  The  selected  studies 
employed  user  satisfaction  or  promoted  its'  employment  as  a 
measurement  tool  for  some  aspect  of  information  system 
effectiveness.  These  included  empirical  studies  on  system 
effectiveness  and  methodological  prescriptions  for  the 
employment  of  user  satisfaction  in  measuring  system 
effectiveness.  These  studies  are  listed  in  Appendix  B. 
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b.  Alternative  Measures   of  Effectiveness 

Appendix  C  lists  the  19  studies  included  in  this 
section.  These  alternative  measures  of  effectiveness  are 
described  in  the  preceding  section. 

c.  Miscellaneous  Literature 

Appendix  F  contains  the  29  remaining  studies 
reviewed.  Included  here  are:  literature  reviews,  essays, 
prescriptive  methodologies,  frameworks  and  tools;  and  10 
articles  not  specifically  dealing  with  effectiveness 
measurement  or  were  too  unclear  as  to  method  or  objective  to 
evaluate  properly. 

B.   REVIEWING  THE  USER  SATISFACTION  STUDIES 

The  40  user  satisfaction  studies  in  Appendix  B  were 
evaluated  with  respect  to  the  following  criteria: 

1.  The  conventional  wisdom  assertion  (CWA) . 

2.  Was  the  Justification  Supported? 

3.  References   rejecting   alternative   measures   of 
effectiveness . 

1.  The  Conventional  Wisdom  Assertion  (CWA) 

The  idea  of  documenting  the  CWA  occurred  after  the 
initial  review  of  all  articles.  The  impetus  came  from  the 
following  observations.  One,  that  there  were  articles  which 
directly  claimed  that  user  satisfaction  was  the  generally 
accepted  or  prevalent  method  for  measuring  IS  effectiveness. 
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Two,   that  often  the  only  evaluation  measure  mentioned  or 

considered  was  user  satisfaction,  i.e.,  possible  alternative 

measures  of  IS  effectiveness  did  not  receive  mention  let  alone 

consideration. 

Determining  which  studies  made  the  CWA  category  was 

then   necessary.   There   were   varying   degrees   of   subtly 

encountered  when  trying  to  apply  the  CWA  classification,  from 

merely  implied  to  outright  claims  of  prevalence  or  general 

acceptance.   The  outright  claims  were  relatively  easy  to 

identify,  as  most  made  the  outright  claim  in  the  Abstract  or 

Introduction.  Some  examples  are: 

The  measurement  of  how  satisfied  a  user  is  with  his  or  her 
information  system  (user  information  system  or  UIS)  has 
become  a  pervasive  measure  of  the  success  or  effectiveness 
of  an  information  system.  (Baroudi  and  Orlikowski,  1988) . 

To  measure  system  success,  researchers  have  mainly  relied 
on  either  of  two  proxies:  user  satisfaction  or  system  use. 
(Barki  and  Huff,  1985) 

User  information  satisfaction  is  the  preferred  indicator 
of  system  success  ...  (Tait  and  Vessey,  1988) 

One  of  the  most  widely  used  methods  of  evaluating  system 
effectiveness  is  the  user  satisfaction  survey."   "...  our 
rather  wholesale  acceptance  of  these  [user  satisfaction] 
evaluation  tools.  (Hufnagel,  1990) 

The   primary   consideration   in   evaluating   Management 
Information  Systems  (MIS)  effectiveness  is  that  it  ^must 
be  evaluated  on  the  criteria  of  importance  to  the  users'  . 
(Lucas,  1978) 

user  satisfaction  with  MIS  . . .  provides  the  most  useful 
assessment  of  system  effectiveness.  (Hamilton  and 
Chervany,  1981) 

Table  I  of  Appendix  E  contains  those  studies  that  were 

identified  as  having  made  the  CWA.  Appendix  D,  Annotated 
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Description  of  User  Satisfaction  Articles,  contains  the 
selected  documentation  evidence  used  for  assigning  the  CWA 
classification  to  a  study.  Included  in  the  description  are 
quotations  and  page  references.  For  outright  claims  that  user 
satisfaction  is  the  prevalent  or  accepted  method  of  measuring 
system  effectiveness,  this  documentation  serves  as  complete 
evidence  for  the  CWA  classification  decision. 

Those  that  only  implied  the  CWA  were  considerably  more 
difficult  to  notice  and  especially  difficult  to  document.  This 
may  seem  contradictory  to  the  notion  of  conventional  wisdom. 
However,  when  taken  to  the  extreme,  conventional  wisdom  is  an 
assumption  by  its  user  and,  as  such,  is  easily  overlooked  and 
difficult  to  pinpoint  by  an  outside  party  searching  for 
specific  documentation.  Additionally,  the  purpose  of  some 
studies  was  to  further  develop  or  test  a  particular  user 
satisfaction  tool.  Generally,  for  these  cases  it  was  often 
necessary  to  examine  the  Introduction,  Methodology  (for 
description  of  the  measurement  tool) ,  and  Conclusions  sections 
to  determine  the  extent  of  an  author's  assumption  regarding 
his  intended  and  actual  employment  of  the  tool  as  a  measure  of 
effectiveness,  with  respect  to  the  CWA. 

The  annotated  description  of  User  Satisfaction 
Articles,  Appendix  D,  contains  relevant  quotations  concerning 
the  CWA.  It  should  be  noted  that  the  quotations  from  some 
studies  do  not  clearly  indicate  a  conventional  wisdom 
statement.  However,  they  do  at  least  suggest  an  attitude  of 
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acceptance  of  the  measure,  which,  when  combined  with  other 

statements  in  the  article,   provided  enough  circumstantial 

evidence  for  it  to  be  included  in  the  CWA  category.  The 

circumstantial  evidence  which  lead  to  the  CWA  judgement  was 

mainly  obtained  from  reading  the  Introduction,  Methodology  and 

Conclusions  sections  with  respect  to  intent,  actual  employment 

and  information  provided  about   the   system  by  the  user 

satisfaction  tool.  The  following  are  some  examples: 

In  information  systems (IS)  research,  user  satisfaction  (or 
user  information  satisfaction)  is  considered  to  be  a  key 
construct  for  assessing  system  performance.  (Barki,  1990) . 

While  acknowledging  the  importance  of  economic  analysis  of 
MIS  value,  researchers  respond  to  the  shifting  emphasis 
form  efficiency  to  effectiveness  by  focusing  either  on  MIS 
usage  or  user  perceived  effectiveness.  (Srinivasan,  1985) 

2.  Justification  Supported 

Justification  Supported  was  the  classification  given 
to  studies  which  properly  applied  user  satisfaction  as  an 
effectiveness  measure.  When  determining  whether  the  employment 
of  user  satisfaction  was  justified  the  criterion  of  this  study 
was  the  degree  to  which  the  tool  was  to  measure  effectiveness. 
Was  it  as  a  "relative"  measure  or  as  an  "absolute"  measure  of 
system  effectiveness?  In  other  words,  a  "better"  or  "more" 
effective  system  rather  than  that  the  system  "was"  or  "was 
not"  successful.  The  choice  of  relative  or  absolute  measure 
was  selected  as  best  describing  the  myriad  of  approaches  to 
employment  user  satisfaction  has  in  the  literature,  from  a 
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single  item  (factor)  measure  of  early  works,  to  multi-factor 
measures  employed  in  a  triangulat ion  method. 

This  study  used  as  its  basis  for  judgement,  arguments 
from  Denzin  (1989)  and  Phillips  (1971)  .  These  were  chosen  for 
their  information  concerning  the  survey  technique,  its  use  and 
limitations . 

Denzin  presents  the  theory  of  symbolic  interactionism 

as  the  perspective  from  which  to  analyze  research  methods. 

Briefly  summarized  the  concept  of  symbolic  interactionism  is: 

If  human  behavior  is  observable  at  two  levels  -  the 
symbolic  and  the  interactional  -  then  central  to 
understanding  such  behavior  is  the  range  and  variety  of 
symbols  and  symbolic  meaning  shared;  communicated  and 
manipulated  by  interacting  selves  in  social  situations. 
Society  contributes  two  essential  elements  that  reflect 
directly  on  concrete  interactions:  (1)  the  symbols;  or 
various  languages  provided  and  communicated  through  the 
socialization  process,  and  (2)  the  concrete  behavioral 
settings  in  which  the  interaction  occurs.  (Denzin,  1989) 

With  respect  to  this  study,  the  symbolic  level  is 
interpreted  as  the  opinion  expressed  by  the  user  regarding  his 
or  her  satisfaction  with  the  system  and  the  interactional 
level  is  that  of  the  observed  or  measured  behavior, 
effectiveness  or  performance,  of  the  user  himself.  Regarding 
the  survey  technique,  Denzin  states  that  "  Users  of  the  survey 
seldom  combine  symbols  with  interaction  ..."  (Denzin,  1989). 

Phillips,     when     discussing     interviews     and 

questionnaires  states  that: 

.  .  .  inferences  about  nonverbal  behavior  which  are  based  on 
nonverbal  responses  are  more  subject  to  distortion  and 
errors  than  are  inferences  about  future  or  past  nonverbal 
behavior  which  are  based  on  the  observation  of  such 
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behavior.  That  is,  it  is  easier  to  xlie  with  words'  than 
to  xlie  with  acts'  (Phillips,  1971) . 

Here  again  the  author  questions  the  ability  of  the 
survey  technique  to  draw  inference  to  nonverbal  behavior, 
i.e.,  action.  Furthermore,  only  some  of  the  user  satisfaction 
authors,  Table  VIII,  have  addressed  questions  of  measurement 
validity  and  reliability.  However,  one  particular  type 
proposed  by  Phillips  that  seems  pertinent  has  not,  convergent- 
descriminant  validity.  This  is  the  ability  to  invalidate 
measures  by  showing  that  they  "correlate  poorly  with  some 
criterion  but  also  by  showing  not  only  that  they  correlate 
highly  with  some  conceptually  simpler  variable,  such  as  the 
tendency  to  respond  true  or  in  a  socially  desirable  manner  at 
all  times."  (Phillips,  1971). 

The  basic  premise  in  the  current  study  is  drawn  from 
arguments  by  Denzin  and  Phillips,  that  is,  that  users 
responses  on  a  user  satisfaction  survey  regarding  information 
system  effectiveness  (opinion)  are  not  the  same  as  measuring 
performance  as  a  result  of  an  information  system.  This  study 
makes  a  differentiation  between  the  relative  measure, 
employment  as  a  descriptor  or  supplementary  measure  of  the 
system  and  the  absolute  measure,  success  or  failure  of  the 
system  as  its  means  of  classifing  opinion  (relative)  and 
action  (absolute)  .  The  delineation  was  made  by  drawing  an 
anology  to  the  opinion-performance  arguements  of  Denzin  and 
Phillips.   Relative  measures  were  viewed  as  anologous  to 


30 


opinion,  i.e.  they  can  enhance  or  supplement  objective  or 
primary  measures  but  cannot  stand  on  their  own.  Absolute 
measures,  on  the  other  hand,  have  the  necessary  and  sufficient 
conditions  to  stand  on  their  own.  The  relative  measures  were 
designated  Justification  Supported  while  the  absolute  were 
not . 

Deciding  if  a  particular  study's  employment  of  user 
satisfaction  was  justified,  this  study  first  looked  at  whether 
the  article  being  reviewed  employed  a  multiple-criteria 
method,  i.e.,  user  satisfaction  was  supplementing  information 
from  another  measurement  technique.  This  ancillary  use  was 
determined  to  be  a  justified  use  of  the  user  satisfaction  tool 
and  classified  as  a  relative  measurement  thus  was  a  case  of 
Justification  Supported.  There  was  considerable  latitude  in 
applying  the  multi-criteria  test.  This  broad  interpretation 
was  because  this  author  was  not  skilled  in  determining  the 
validity  of  the  various  research  methods  encountered  and 
wished  to  be  as  conservative  as  possible  in  assigning 
restrictions  to  the  method,  thereby  providing  the  researchers 
the  widest  possible  latitude  in  their  employment  of  the  user 
satisfaction  tool. 

Studies  which  used  user  satisfaction  as  the  sole  means 
of  measuring  the  system  effectiveness  constituted  the  other 
major  category.  It  was  necessary  to  determine  the  use  of  the 
user  satisfaction  tool's  value,  was  it  for  relative 
qualification  or  judgement  of  the  system  or  was  it  for  an 
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absolute  quantification  or  description  of  the  system's  success 

or  failure.  This  was  a  particularly  difficult  assessment  to 

make.  As  with  the  CWA,  some  authors  stated  directly  how  they 

intended  to  employ  the  tool,  while  others  had  conflicting 

statements  in  their  articles  which  required  analyzing  the 

intent  and  conclusions  drawn  from  the  measure's  employment. 

The   following   are   examples   of   the   basic   Justification 

Supported  categories: 

Justification  Supported:  Yes,  relative  measure. 

concerned  only  with  identifying  difference  in  the 
affective  response  patterns  of  the  two  groups  of 
performers  (as  opposed  to  arriving  at  a  'true'  measure  of 
value.  (Hufnagel,  1990) 

Justification  Supported:  No,  absolute  measure. 

The  measurement  of  how  satisfied  a  user  is  with  his  or  her 
information  system  (user  information  satisfaction  or  UIS) 
has   become   a   pervasive   measure   of   the   success   or 
effectiveness  of  an  information  system.   (Baroudi  and 
Orlikowski,  1988) 

Justification  Supported:  Yes,  multiple  criteria. 

The  purpose  of  this  study  was  to  apply  four  methodologies 
to  help  understand  and  measure  current  information  systems 
.  .  .  The  objective  is  to  recommend  an  easy  to  use 
instrument  for  improving  the  effectiveness  of  information 
systems.  (Evans  et  al.,  1988) 

Table  I  of  Appendix  D  contains  the  determination  data 

on  Justification  Supported.  Additionally  it  contains  the  page 

number  where  relevant  information  leading  to  the  judgement  is 

located.  Appendix  D  contains  selected  quotations  which  either 

provide  direct  evidence  or  substantial  circumstantial  evidence 
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documenting  the  study's  documentation  Justification  Supported 
decision . 

Another  category  of  data  in  this  thesis  was  whether 
there  was  a  specific  reference  to  rejection  of  an  alternative 
measure  (s)  of  system  effectiveness,  see  Table  V,  Appendix  E. 
All  that  was  required  was  for  a  study  to  include  somewhere  in 
its  discussion  and  in  its  list  of  references  an  alternative 
measure.  The  point  being,  that  there  was  an  attempt  made  at 
selecting  an  appropriate  method  to  measure  system 
effectiveness . 
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IV.  FINDINGS 

Appendix  D  contains  an  annotated  description  of  each  of 
the  40  user  satisfaction  studies  evaluated  here.  Appendix  E 
contains  the  detailed  tables  of  findings  for  each  aspect  of 
this  study.  They  are  listed  in  chronological  order  by  date  of 
publication . 

Studies  in  Appendix  B  employed  a  form  of  user  satisfaction 
measure  to  evaluate  the  effectiveness  of  an  information 
system. 

In  Appendix  C,  the  studies  used  an  alternative  measurement 
tool  to  evaluate  system  effectiveness. 

A.   CONVENTIONAL  WISDOM  ASSERTION 

The  first  question  for  which  an  answer  was  sought  was  what 
number  of  articles  assume  user  satisfaction  is  currently  the 
conventional  wisdom  or  prevalent  method  employed  by  the  IS 
community  when  measuring  the  effectiveness  of  an  information 
system. 

Table  I  of  Appendix  E  condenses  out  of  Appendix  B  the 
primary  references  cited  by  each  author  for  choosing  user 
satisfaction  as  a  surrogate  measure  of  system  effectiveness 
and  for  those  with  none  listed  the  conventional  wisdom 
assertion  (CWA)  was  assigned.  Table  II  shows  that  the  CWA  is 
the  only  reference  for  45%  of  the  MIS  studies  that  employ  user 
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satisfaction  as  a  measurement  tool.  This  implies  that  MIS 
researchers  presuppose  not  only  that  user  satisfaction  is  the 
preferred  method  of  measuring  MIS  effectiveness  but  that  it 
has  also  been  proven  and  accepted  as  such  by  the  MIS 
community.  This  is  certainly  not  the  case  as  shown  by  Appendix 
C,  which  lists  19  studies  that  use  alternative  methods  of 
measuring  an  information  system's  effectiveness.  Additionally, 
it  contains  several  references  in  MIS  literature  rejecting 
user  satisfaction  as  an  absolute  or  single  value  measurement 
of  a  system's  effectiveness  (Ahituv,  1980;  Marsden  and 
Pingrey,  1988;  Hufnagel,  1990)  on  theoretical  or 
methodological  grounds. 

Additional  distillation  of  the  data  in  Table  I  at  years 
1977  (Pearson)  and  1983  (Bailey  and  Pearson)  provides  further 
insight  into  the  genesis  and  evolution  of  the  CWA.  In  1977 
Pearson's  dissertation  developed  a  tool  for  measuring  user 
satisfaction.  In  1983  Bailey  and  Pearson  further  refined  the 
tool  in  a  published  article  (Bailey  and  Pearson,  1983)  .  Evans 
et  al.  (1988)  state  that  "Instruments  to  measure  user 
satisfaction  are  scarce."  Most  of  those  that  existed  prior  to 
an  empirically  derived  methodology  developed  by  Pearson  have 
been  heavily  criticized.  Now,  examining  the  percentage  of  the 
CWA  in  studies  from  1977  forward  and  1983  forward  it  appears 
that  the  emergence  of  the  CWA  has  it's  origins  with  Pearson's 
1977  dissertation  and  it's  widespread  practice  with  the 
publishing  of  the  1983  Bailey  and  Pearson  study.  There  is  also 
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an  increase  in  the  number  of  empirical  studies  evaluating  IS 
(Alavi  et  al . ,  1989) . 

Studies  with  a  CWA  of  user  satisfaction  have  grasped  this 
tool  as  the  an  easy  and  reliable  way  to  measure  system 
effectiveness.  Often  they  have  ignored  the  limitations  stated 
by  Bailey  and  Pearson  concerning  its  use.  Bailey  and  Pearson 
propose  that  it  be  used  for  "relative"  evaluation,  i.e. 
comparison  of  and  between  systems.  Many  authors  making  the  CWA 
imply  and  draw  conclusions  of  user  satisfaction's  absolute 
measurement  of  system  effectiveness. 

B.   JUSTIFICATION  SUPPORTED 

Table  III  contains  the  studies  of  Appendix  B  which  have  a 
justification  for  employment  of  user  satisfaction  that  is 
supported  by  the  reference  literature,  i.e.,  that  the  user 
satisfaction  tool  was  used  properly  according  to  a  cited, 
developed  methodology  and  subsequent  conclusions  limited  to 
the  theoretical  restrictions  of  the  tool.  The  judgement 
supported  perspective  shows  that  over  40%  of  the  studies 
employing  user  satisfaction  have  incorrectly  applied  the  tool. 
The  most  common  error  was  to  wrongly  imply  an  absolute  value 
of  system  effectiveness  from  the  level  of  user  satisfaction, 
i.e.,  a  successful  or  unsuccessful  system,  rather  than  a 
relative  measure,  which  is  a  comparison  between  alternatives 
or  situations. 
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Only  19  of  the  40  user  satisfaction  studies  properly 
applied  the  user  satisfaction  tool.  Those  that  did  limited  its 
range  of  measure  to  some  aspect  of  relative  effectiveness 
rather  than  employing  it  as  an  absolute  measure. 

Eight  of  the  19  are  multiple  criteria  evaluations  where 
user  satisfaction  was  employed  as  one  of  several  effectiveness 
measures.  The  multiple  criteria  approaches  in  general  centered 
around  "the  relationship  between  satisfaction  and  bottom  line 
indicators  of  user  and  organization  performance"  (Bailey  and 
Pearson,  1983)  .  These  bottom  line  indicators  are  what  this 
study  considers  absolute  measures  of  effectiveness 
(performance)  . 

When  user  satisfaction  was  employed  as  a  single  item 
measure  it  was  most  commonly  used  as  a  "tool  for  improving 
system  design"  (Bailey  and  Pearson,  1983),  which  is 
supplementary  or  descriptive  and  thus  classified  as  a  relative 
measure  by  this  study. 

Further  examination  of  the  studies  that  use  the  CWA  shows 
that  66%  of  them  have  also  incorrectly  applied  the  tool.  The 
implication  here  is  that  not  only  have  researchers  not 
evaluated  other  methods  of  measurement  as  evidenced  by  the 
CWA,  but  that  they  have  also  not  evaluated  the  user 
satisfaction  tool  itself  and  understood  its  limitations  and 
restrictions  in  particular  those  citing  Bailey  and  Pearson. 
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C.  REFERENCES  TO  REJECTION  OF  ALTERNATIVE  MEASURES 

Table  V  applies  a  breakdown  similar  to  both  CWA  and 
Justification  Supported  with  respect  to  studies  which  actually 
reference  an  article  which  supports  the  rejection  of 
alternative  measures  to  user  satisfaction.  Common  statements 
such  as  "it  is  too  difficult"  or  "too  costly"  without 
reference,  forms  another  category  of  conventional  wisdom, 
i.e.,  that  alternative  methods  cannot  be  used  for  measuring 
system  effectiveness.  MIS  authors  using  user  satisfaction  have 
polarized  their  view  of  MIS  effectiveness  measurement  to  the 
point  that  it  consists  solely  of  the  user  satisfaction  tool. 

D.  REJECTION  OF  USER  SATISFACTION 

Various  articles  in  MIS  literature  have  rejected,  whole  or 
in  part,  user  satisfaction  as  a  surrogate  measure  of  system 
effectiveness.  Hall  and  Lincoln  (1976)  "considered  the 
possibility  of  measuring  user  satisfaction  as  a  surrogate 
measure  for  information  utility  . . .  However  it  was  felt  that 
an  index  of  user  satisfaction  would  not  have  provided  senior 
BAC-CAD  management  with  the  means  to  compare  the  effectiveness 
of  information  systems  with  that  of  other  company  resources." 
This  early  rejection  of  the  measure  conceptualizes  the 
ultimate  purpose  of  any  absolute  evaluation  measure,  which  is 
an  overall  assessment.  The  value  of  which  is  either  for 
judging  the  success  or  failure  of  the  system  or  as  a  common 
measure  to  compare  one  system  to  another.  In  either  case  the 
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value  of  the  measure  is  complete  in  itself,  i.e.,  it  can  stand 
alone  and  convey  valuable  information,  for  justifying  company 
resources  or  comparing  completely  separate  information 
systems . 

The  other  use  to  which  an  evaluation  measure  is  put  is  to 
compare  the  relative  value  within  a  single  item  context,  i.e., 
better/worse  rather  than  successful/ failure .  Here  again, 
critics  have  found  fault  with  user  satisfaction  on  both 
theoretical  and  methodological  grounds.  Many  reject  it  saying 
"single  item"  measures  are  inadequate  (Larker  and  Lessig, 
1980;  Hamilton  and  Chervany,  1981;  Barki  and  Huff,  1985) . 
Additionally,  others  (Matlin,  1979;  Markus  and  Robey,  1983; 
and  Jarvenpaa  et  al.,  1985)  argue  that  user  satisfaction 
measures  often  lack  theory  and  that  the  results  are  subject  to 
influence  by  other  factors  within  the  system  and  therefore  it 
is  an  inconsistent  measure  and  a  meaningless  tool  for 
evaluation . 
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V.  CONCLUSIONS 

The  evolution  of  user  satisfaction  as  a  surrogate  measure 
for  IS  effectiveness  suffers  from  two  deficiences.  The  first 
is,  that  in  45  percent  of  the  studies  reviewed,  it  was  the 
only  method  considered,  which  this  study  has  labled  the 
conventional  wisdom  assertion  (CWA) .  The  second  stems  in  part 
from  the  CWA,  that  in  over  65  percent  of  the  studies  the  tool 
was  improperly  applied. 

The  MIS  community's  desire  for  credibility  as  a  research 
field  has  led  in  part  to  the  CWA.  A  credible  user  satisfaction 
measurement  tool  was  lacking  before  that  introduced  by  Pearson 

(1977)  and  developed  by  Bailey  and  Pearson  (1983) .  This  prior 
void  and  the  lack  of  empirical  studies  in  MIS  research  (Alavi 
et  al.,  1989)  contributed  in  large  part  to  a  blind  acceptance 
of  the  user  satisfaction  tool  as  a  surrogate  measure  for 
system  effectiveness  by  those  making  the  conventional  wisdom 
assertion.  Many  in  the  MIS  community  grasped  the  Bailey  and 
Pearson  tool  without  regard  to  its  foundation,  often 
improperly  applying  it.  By  failing  to  question  the  newly 
developed  tool  and  even  failing  to  understand  its  limitations, 
those  making  the  CWA  have  done  a  disservice  to  the  MIS 
community  and  MIS  as  a  whole.  Citing  Keen  (1980)  Alavi  et  al. 

(1989)  in  their  review  of  MIS  research  say  "Even  emerging 
fields  such  as  MIS  have  a  sense  of  history;  those  who  cut 
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themselves  off  from  curiosity  and  reflection  are  likely  to 
atrophy . " 

Additionally,  the  MIS  community' s  need  for  empirical 
research  (Alavi  et  al.,  1989)  facilitated  the  acceptance  of 
user  satisfaction  as  an  empirical  tool.  There  were  no 
instances  of  the  CWA  in  MIS  studies  prior  to  Pearson's  1977 
dissertation  and  by  Bailey  and  Pearson's  1983  study  57  percent 
of  those  that  employed  user  satisfaction  made  the  CWA.  This 
highlights  the  blind  acceptance  of  it  by  many  researchers  in 
the  MIS  community. 

The  second  problem  is  that  of  improper  application. 
Basically  the  user  satisfaction  tool,  as  developed  by  Pearson 
and  modified  by  others  (Bailey  and  Pearson,  1983;  Ives  et  al., 
1983)  is  intended  to  measure  satisfaction  as  an  indicator  of 
the  relative  effectiveness  of  an  information  system  by  means 
of  "his/her  reaction  to  the  [user  satisfaction]  factor 
relative  to  the  perceived  information  requirements"  (Bailey 
and  Pearson,  1983) .  If  the  user  satisfaction  value  is  higher 
for  one  case  than  another  the  higher  valued  system  will  be 
more  effective  than  a  lower  valued  one,  whether  it  is  the  same 
system  which  was  or  will  be  modified  or  a  totally  sepatate 
one.  It  is  not  able  to  evaluate  the  absolute  effectiveness  of 
an  information  system.  That  would  require  a  comparison  between 
inputs  and  outputs  of  the  system (s) .  As  Marsden  and  Pingrey 
(1988)   point  out   "the  key  end  user,   the  firm,   will  be 
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^satisfied'   when   its   investment   decision,   that   of   the 
information  system,  contributes  to  the  success  of  the  firm." 

Therefore,  if  properly  applied  employing  user  satisfaction 
as  a  surrogate  measure  for  system  effectiveness  may  indicate 
"more"  or  "less"  effective  systems.  However,  40  percent  of  all 
user  satisfaction  studies  reviewed  and  57  percent  of  those 
which  were  conducted  in  1983  and  later  (Bailey  and  Pearson) 
improperly  applied  the  tool.  This  improper  application  is  due 
to  its  employment  as  an  absolute  measure  of  system 
effectiveness.  Combined  with  the  conventional  wisdom 
assertion,  the  implication  is  that  MIS  research  using  user 
satisfaction  is  on  uncertain  ground.  Not  only  is  the 
conventional  wisdom  assertion  made  by  many  authors  (tablel) 
not  justified  but  when  employed  user  satisfaction  is 
improperly  applied  nearly  half  the  time.  The  researchers  have 
not  only  accepted  the  surrogate  unquestioningly  but  they  have 
also  not  bothered  to  understand  and  correctly  apply  it.  This 
has  a  negative  effect  on  the  MIS  community  which  is  seeking 
credibility  as  a  scientific  research  field  (Alavi  et  al., 
1989)  and  for  the  justification  of  the  financial  investments 
being  made  in  information  systems  (Fortune,    26  May  1990)  . 

The  intent  of  this  study  was  not  to  determine  the 
scientific  validity  of  user  satisfaction  as  a  surrogate 
measure  of  effectiveness.  It  was  however,  intended  to  shed 
some  light  on  its  evolution  and  use  in  MIS  literature. 
Examining  studies  measuring  system  effectiveness   it  was 
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determined  not  only  that  methods  other  than  user  satisfaction 
are  "alive  and  well,"  thus  dispelling  the  conventional  wisdom 
assertion,  but  also  that  user  satisfaction  may  be  the 
"hobgoblin"  of  effectiveness  measurement  (Soergel,  1976) . 
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VI .  RECOMMENDATIONS 

The  role  of  MIS  research  is  to  increase  our  understanding 
of  information  systems,  thereby  allowing  prediction  of 
outcomes  and  consequences  of  the  various  factors  influencing 
an  information  system.  To  do  so  researchers  must  develop 
theories  with  which  to  guide  research  and  research 
methodologies  (Alavi  et  al.,  1989)  which  provide  a  systematic 
investigation  of  these  factors.  The  focus  of  this  study  was  on 
the  measurement  of  effectiveness  of  information  systems.  In 
particular,  the  evolution  and  use  of  user  satisfaction  as  a 
surrogate  measure  of  effectiveness.  At  issue  was  the 
employment  of  user  satisfaction,  an  opinion,  as  a  measure  of 
effectiveness,  an  action. 

The  articles  read  for  this  study  and  the  subsequent 
findings  suggest  several  things  regarding  the  measurement  of 
effectiveness  of  information  systems.  To  begin  with,  the  MIS 
community  must  define  a  standard  framework  of  information 
system  effectiveness.  The  wide  variety  of  effectiveness 
definitions  encountered  made  it  difficult  to  assess  the 
validity  of  any  given  measure.  Additionally,  a  "systematic 
investigation"  is  difficult  in  that  findings  are  not 
replicated  and  verified,  and  methodologies  are  not  advanced  by 
refinement . 
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Next,  theories  should  be  developed  and  tested.  The  fact 
that  almost  half  of  the  user  satisfaction  studies  applied  the 
CWA  and  that  40%  improperly  applied  the  tool  indicates  that 
the  field  is  not  thinking  about  the  theory  behind  the 
relationship.  The  difficulty  that  the  academic  field  of 
management  science  has  found  linking  satisfaction  with 
performance  should  draw  into  question  an  analogous 
relationship  in  MIS  between  user  satisfaction  and 
effectiveness . 

Additionally,  social  scientists  have  questioned  the 
validity  of  the  survey  as  a  research  method  and  in  particular, 
as  an  accurate  measure  of  behavior.  Less  than  a  quarter  of  the 
user  satisfaction  studies  did  any  validity  checks  and  only  one 
questioned  the  validity  of  user  satisfaction  as  a  surrogate 
for  effectiveness  by  testing  for  other  possibilities.  Instead 
the  validity  checks  merely  sought  to  find  a  correlation 
between  user  satisfaction  and  another  attitudinal  variable. 

The  ubiquitous  statements  in  the  literature  that 
intangibles  cannot  be  accurately  measured,  are  too  difficult, 
too  time  consuming  or  too  costly  to  obtain,  often  without 
reference  suggest  a  general  attitude  exists  in  the  MIS 
community  regarding  effectiveness  measurement.  Combined  with 
the  fact  that  the  MIS  community  seems  to  have  embraced  the  use 
of  a  subjective  surrogate  for  the  same  ends,  it  makes  one 
wonder  about  the  motivation  or  competence  behind  the  research. 
Are  we  further  extending  our  knowledge  of  information  systems 
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or,  is  it  an  easy  way  to  publish  and  gain  tenure?  Was  the 
research  strategy  selected  the  best  choice  or  the  easiest? 

The  MIS  community  needs  to  provide  valuable  information 
concerning  information  systems  to  the  user  community  so  that 
they  may  invest  their  capital  resources  more  effectively  in 
information  systems.  As  Marsden  and  Pingrey  (1988)  point  out 
"the  key  end  user,  the  firm,  will  be  ^satisfied'  when  its 
investment  decision,  that  of  the  information  system, 
contributes  to  the  success  of  the  firm." 
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APPENDIX  B.  LIST  OF  STUDIES  EMPLOYING  USER  SATISFACTION  AS 

EFFECTIVENESS  MEASURE 


STUDY 

YEAR 

CWA 

JUSTIFICATION 
SUPPORTED 

REFERENCE 

REJECTING 

ALTERNATIVES 

i 

Barxi 

1 

90 

Y 

*p.  408 

N 

*p.  402 

„ 

2. 

Burkhard 

90 

N 

<MC) 

Y 

p.  421 

NA   (MC) 

3. 

Conrath/Mignen 

90 

Y 

p.  7 

Y 

p.  16 

N 

4. 

Huf nagel 

90 

Y 

p.  437 

Y 

p.  441,  437 

N 

5. 

Baroudi/Orl ikowski 

90 

Y 

p.  44 

N 

p.  44-45 

N 

6. 

Doll/Torkadek 

90 

N 

N 

p.  260 

Y 

7. 

Evans  et  al. 

90 

N 

(MC) 

Y 

p.  228,236 

NA   (MC) 

8. 

Gallupe/DeSanctis 

90 

\ 

(MC) 

Y 

p.  284 

NA   (MC) 

9. 

Gumaraes /Gupta 

90 

Y 

p.  18 

N 

p. 18 

\ 

10. 

Tait/Vessey 

90 

Y 

p.  95-96 

N 

P.  95-96 

N 

11. 

Miller/Doyle 

90 

Y 

p.  108 

N 

p.  107-108 

N 

12. 

Aldag/Power 

90 

N 

(MC) 

Y 

p.  579,  584 

NA   (MC) 

13. 

Baroudi/01 son/I ves 

90 

Y 

P.  233 

Y 

p.  233 

1 
N 

14. 

Franz/Robey 

90 

Y 

p.  335 

Y 

N 

15. 

Franz/Robey/Koeblitz 

90 

Y 

p.  32 

Y 

p.  32 

N 

16. 

Martin/Trumbley 

90 

N 

(MC) 

Y 

p.  14 

NA   (MC) 

17. 

Barki/Huf f 

90 

Y 

p.  262 

N 

p.  262 

Y   p.  262 

18. 

Kogue/Watson 

90 

N 

p.  206 

N 

p.  210-211 

NA  (?) 

19. 

Srinivason 

85 

Y 

p.  243 

Y 

p.  243 

V 

20. 

Bailey/Pearson 

83 

Y 

p.  530 

Y 

p.  530 

N 

21. 

Ives/Olson/Baroudi 

83 

Y 

P.  785 

N 

p.  791,  792 

Y   p.  785 

22. 

King/Epstein 

83 

N 

Y 

p.  34,43 

NA   (MC) 

23. 

Mahmood 

83 

Y 

p.  171 

N 

p.  171 

N 

24. 

Epstein/King 

82 

N 

N 

p.  255 

NA 

25. 

Ginzbrerg 

81 

Y 

p.  467 

N 

p.  463 

N 

26. 

Hamilton/Chervany 

81 

Y 

p.  79 

N 

p.  84 

NA  (MC) 

27. 

Larker/Lessig 

80 

Y 

p.  121 

N 

p.  121 

N 

28. 

Deese 

79 

NA 

Y 

p.  66 

NA 

29. 

Nueman/Segev 

79 

N 

p.  35 

V 

p.  35 

N 
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STUDY 

YEAR 

CWA 

JUSTIFICATION 
SUPPORTED 

REFERENCE 

REJECTING 

ALTERNATIVES 

30. 

Robey 

79 

N 

Y 

p.  534,  537 

N 

31. 

King/Rodriguez 

78 

N 

Y 

p.  45,  50 

NA   (MC) 

32. 

1  jcas 

78 

N 

N 

p.  72 

Y   p.  69 

33. 

Robey /Zel lar 

78 

N 

Y 

p.  71,  73 

N 

34. 

Dickson /Ser.r.  /Chervany 

77 

N   (MC) 

Y 

NA   ( MC ) 

35. 

Edstrom 

77 

N 

N 

p.  589 

N 

36. 

Schewe 

76 

N 

Y 

p.  589 

N 

37. 

Gallagher 

74 

N 

Y 

p.  52 

Y 

38. 

Lucas 

74 

N 

Y 

p.  210-211 

NA   (MC) 

39. 

Swanson 

74 

N 

Y 

p.  180 

N 

40. 

Powers/Dickson 

73 

N 

Y 

p.  150 

NA   (MC) 

*  denotes  the  page  number  in  the  study  where  statements 
supporting  these  classifications  are  located 
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APPENDIX  C.  LIST  OF  STUDIES  EMPLOYING  ALTERNATIVE  METHODS  OF 

EFFECTIVENESS  MEASURE 


STUDY 

YEAR 

EFFECTIVENESS 
MEASUREMENT  TOOL 

REFERENCE 
U  SAT 

REJECTING 
OR  AH 

1. 

Goidstein/Cho/Zack 

90 

Knowledge 

1 

2. 

Mor.ey/Tromp/Wegner 

88 

Value  analysis 

Y   p.  223 

3. 

Marsden/Pingrey 

88 

Profit  max 

y  p-  76 

4. 

Kapsaies 

86 

Cost     avoidance 
(product ivi  ty ) 

N 

5. 

Lincoln 

86 

CBA 

Y   p.  26 

6. 

Sullivan/Secrest 

85 

Optimization 
(profit) 

N 

. 

Engl ander /Engl  an der 

85 

Cost  savings 

N 

8. 

Snitkin/King 

86 

User  attitude 

\ 

9. 

Yovitz/Foulk 

85 

Value  analysis 

N 

10. 

Cron/Soboi 

83 

Prof itabi  ii ty 

Y   p.  172 

11. 

Chandler 

82 

Multiple  cirteria 

NA 

12. 

Edelman 

81 

CBA 

Y   p.  5 

13. 

Altmeyer/Bozeman 

81 

User  attitude 

N 

14. 

Mansour/Watson 

80 

Multiple  criteria 

NA 

15. 

Ahitau 

80 

Utility 

Y   p.  61 

16. 

Maltin 

79 

Value  analysis 

N 

17. 

Radeccki 

76 

N 

18. 

Hall/Lincoln 

76 

CBA 

Y   p.  266 

19. 

Lucas 

75 

Useage 

N 
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APPENDIX  D.  ANNOTATED  DESCRIPTION  OF  USER  SATISFACTION 

STUDIES 


Barki,  H.  "Determinants  of  User  Satisfaction  Judgements  in 
Information  Systems, "  Proceedings  of  the  Twenty — third  Annual 
Hawaii  Conference  on  System  Sciences,  Vol.  VII,  (1990),  pp. 
408-417. 

Attribution,  expectation,  disconf irmation  and  performance 
is  determinant  of  user  satisfaction  judgements. 

"performance  is  much  more  dominate" 

CWA:        "In  information  systems  (IS)  research,  user 

p.  408      satisfaction  (or  user  information  satisfaction)  is 

considered  to  be  a  key  construct  for  assessing 

system  performance." 

JS:         "User  satisfaction  is  also  used  by  IS  professionals 

in 

p.  408  evaluating  the  performance  of  specific 
information  system  applications  or  the  overall 
information  system  in  an  organization." 

"an  implied  equivalence  between  user  evaluations 
of  system  performance  and  their  feelings  of 
satisfaction ...  If  such  an  equivalence  exists,  then 
measuring  satisfaction  as  a  surrogate  for  system 
performance  or  effectiveness  can  be  meaningful." 

RR  Alt:     None 
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Burkhard,  D.L.,  "Examination  of  Factors  Contributing  to  the 
Success  of  the   Implementation  of  Computer-Aided  Software 
Engineering   Technology,  "  Proceedings      of     the      Twenty-third 
Annual   Hawaii    Conference   on   System  Sciences,    Vol.  VI,  (1990), 
pp.  418-427. 

"A  proposed  model  consisting  of  variables  that  potentially 
influence  the  development  and  maintenance  processes  is 
evaluated  in  respect  to  several  performance  measures." 

CWA:         No 

"Satisfaction  with  the  system  has  been  used  as  a 
surrogate  for  IS  effectiveness  (Mahmood  1985- 
1986) " 

NA  -  M.C. 


JS: 

P- 

421 

RR 

Alt 

Conrath,  D.W.,  and  O.P.  Mignen,  "What  is  Being  Done  to 
Measure  User  Satisfaction  with  EDP/MIS, "  Information    and 
Management,    Vol.  19,  (1990),  pp.  7-19. 

If  and  how  organizations  are  measuring  user 
satisfaction  what  do  they  perceive  as  benefits  of 
doing  so. 

not  many  organization  appear  to  do  it  (measure  user 

satisfaction) 

those  that  do  value  it  substantially 

CWA:        "There  is  substantial  literature  that  suggests  it 
p.  7        is  important  to  measure  user  satisfaction  with 
EDP/MIS  products  and  services." 

JS:         (1)  to  try  to  improve  the  quality... 

p.  16       (2)  to  increase  the  degree... 

(3)  to  have  better  relations  with  users... 

(4)  to  determine  the  level  of  sat.  of  users... 

RR  Alt:     None 


59 


Hufnagel,  E.M.,  "User  Satisfaction  -  Are  We  Really  Measuring 
System  Effectiveness?"  Proceedings    of   the    Twenty— Third 
Annual    Hawaii    Conference    on    System   Sciences,    Vol.  VI, 
(1990),  pp.  437-446. 

Examines  whether  "user  perception  regarding  an  IS 
provide  an  accurate  measure  of  system  quality" 

"The  pattern  of  causal  reasoning  observed  here  raise 
serious  questions  about  the  validity  of  employing  user 
sat.  ratings  as  measures  of  system  effectiveness." 


CWA:        "One  of  the  most  widely  used  methods  of 
p.  437      evaluating  system  effectiveness  is  the  user 
satisfaction  survey." 

JS:         "Sure  the  hypothesis  being  tested  (H3)  is 
p.  441      concerned  only  with  identifying  the  differences 
in  the  affective  response  patterns  of  two  groups 
of  performance  (as  opposed  to  arriving  at  a 
"true"  measure  of  value) . " 

RR  Alt:     None 


Baroudi,  J.J.,  and  W.J.  Orlikowski,  "A  Short  Form  Measure  of 
User  Information  Satisfaction:  A  Psychometric  Evaluation  and 
Notes  on  Use, "  Journal    of  Management    Information   Systems, 
Vol.  4,  No.  4,  (Spring  1988),  pp.  44-59. 

Develops  short  form  measure  of  user  satisfaction 
from  B/P 

CWA:        "The  measurement  of  how  satisfied  a  user  is  with 
p.  44       his  or  her  information  system  (user  into 

satisfied  or  unsatisfied)  has  become  a  pervasive 
measure  of  the  success  or  effectiveness  of  an 
information  system." 

JS :         "has  become  a  pervasive  measure  of  the  success  or 

p  44-45      effectiveness  of  an  information  system" 

RR  Alt:     None 
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Doll,  W.J.  and  G.  Torkzadeh,  "The  Measurement  of  End  User 
Computing  Satisfaction, "  Management    Information    Systems 
Quarterly,     (June  1988),  pp.  259-274. 

instrument  for  measuring  end  user  satisfaction 

CWA:         No 

"End  user  computing  satisfaction  (EUCS)  is  a 
potentially  measurable  surrogate  for  utility  in 
decision  making." 

"decision  analysis  approach  is  generally  not 
feasible  (Gallagher,  1974,  Nolan  and  Seward, 
1974) " 


JS 

p. 

259 

RR 

Alt: 

P. 

259 

Evans,  et    al . ,    "An  Instrument  for  Measuring  Effectiveness  of 
Information  Systems,"  Computers   and  Engineering,    Vol.  14, 
No.  3,  (1988),  pp.  227-236. 

Multi  Criteria  approach  to  understand  and  measure 

information  systems 

information  flows 

user  needs 

utility  of  information 

user  satisfaction 
alone  and  in  combination  are  useful  for  improving 
information  systems 

CWA:        No 

JS:         "After  identifying  dissatisfied  individuals  on 
p.  234-235  the  Pearson  NSI  matrix,  studying  results  of  the 
printouts  for  individuals  from  the  Krobock 
Methodology  can  help  to  show  where  improvements 
can  be  made  in  specific  sources  of  information  to 
increase  the  satisfaction  of  these  individuals." 

p.  228      "alone  and  in  combination  are  useful  for 
improving  information  systems" 

RR  Alt:     NA 


61 


Gallupe,  R.B.,  and  G.  DeSanctis,  "Computer-Based  Support  for 
Group  Problem  Finding:  An  Experimental  Investigation," 
Management    Information    System   Quarterly,    Vol.  12,  No.  2, 
(June  1988),  pp.  277-296. 

Examines  Group  DSS  on  group  discussion  quality  and 
individual  perceptions 

"confidence  and  satisfaction  . . .  lower  in  GDSS  supported 
groups  than  in  non-supported  groups" 

CWA :        NA 

JS :         "dependent  variables  were  the  discussion 
p.  284      performances  and  perceived  reactions 

RR  Alt:      NA 


Gurmaraes,  T . ,  and  Y.P.  Gupta,  "Measuring  Top  Management 
Satisfaction  with  the  MIS  Department,"  Omega    International 
Journal    of  Management    Science,    Vol.  16,  No.  1,  (1988),  pp. 
17-24. 

"questionnaire  to  measure  top  management's  satisfaction 
with  their  MIS  department" 

CWA:        "researchers  have  traditionally  used  measures  of 
p.  18      user  information  satisfaction  instead" 

JS:        "How  should  one  assess  overall  MIS  department 
p.  18       performance?" 

RR  Alt:     "Many  of  the  variables  involved  in  assessing  an 

p.  18      MIS  department's  importance  are 

intangible. . .Because  of  the  difficulty  of 
evaluating  the  economics  of  information  systems." 


62 


Tait,  P.,  and  I.  Vessey,  "The  Effect  of  User  Involvement  on 
System  Success:  A  Contingency  Approach,"  Management 
Information    Systems   Quarterly,     (March  1988),  pp.  91-108. 

"Path  analysis  was  used  to  investigate  both  the  direct 
effects  of  the  contingent  variables  on  system  success 
and  the  effect  of  user  involvement." 

CWA:        "assessing  the  satisfaction  of  the  user  with  the 
pp.  95-96   information  provided  by  the  system  is  probably 
the  most  common  approach  to  measuring  system 
success  .  .  .  user  information  satisfaction  is  the 
preferred  indicator  of  system  success" 

JS :         "information  satisfaction  is  the  preferred 
pp.  95-96   indicator  of  system  success  and  is  the  measure 

used  in  this  study  to  define  successful  systems 

development" 

RR  Alt:     None 


Miller,  J.,  and  B.A.  Doyle,  "Measuring  the  Effectiveness  of 
Computer-Based  Information  System  in  the  Financial  Sector, " 
Management    Information    Systems   Quarterly,    Vol.  11,  No.  1, 
(March  1987),  pp.  107-124. 

instrument  for  measuring  information  system' s 
effectiveness 

CWA:        "The  instrument  used  in  this  research  consists  of 
p.  108      38  items,  24  of  which  derive  from  Pearson,  12 
from  Alloway . " 

JS:        "the  effectiveness  of  the  overall  information 
p.  107     system's  effort  ...  is  evaluated  using  an 
instrument  based  on  previous  work" 

RR  Alt:     No 


63 


Aldag,  R.J.,  and  D.J.  Power,  "An  Empirical  Assessment  of 
Computer  Assisted  Decision  Analysis, "  Decision  Sciences, 
Vol.  17,  No.  14,   (Fall  1986),  pp.  572-588. 

lab  study  with  multiple  criteria 

CWA:         NA 

"Individuals  ...  will  exhibit  more  confidence  in, 
and  satisfaction  with" 

NA 


JS 

p. 

576 

RR 

Alt 

Baroudi,  J.J.,  M.H.  Olson  and  B.  Ives,  "An  Empirical  Study 
of  the  Impact  of  User  Involvement  on  System  Usage  and 
Information  Satisfaction,"  Communications    of   the   ACM,    Vol. 
29,  No.  3,  (March  1986),  pp.  232-238. 

"that  user  involvement  leads  to  system  usage  and/or 
information  satisfaction" 

CWA:        "user  satisfaction.  Generally  these  constructs 
p.  233      are  assumed  to  be  indirect  indicators  of  improved 
decision  making  performance" 

JS:         See  CWA 

RR  Alt:     None 


Franz,  C.R.,  and  D.  Robey,  "Organizational  Context,  User 
Involvement,  and  the  Usefulness  of  Information  Systems," 
Decision    Sciences,    Vol.  17,  No.  3,  (Summer  1986),  pp.  329- 
355. 

user  involvement  is  positively  related  to  user's 
perceptions  of  system  usefulness 

CWA:        cites  Bailey  and  Pearson  (1983)  and  Ives,  Olson 
p.  335      and  Baroudi  (1983) 

JS:         "related  positively  to  user's  perceptions" 
p.  329 

RR  Alt:     None 


64 


Franz,  C.R.,  D.  Robey,  and  R.R.  Koeblitz,  "User  Response  to 
an  Online  Information  System:  A  Field  Experiment," 
Management    Information    Systems    Quarterly,     (March  1986),  pp. 
29-42. 

"impact  of  information  systems  on  user  attitudes  and 
perceptions  of  job  characteristics" 

CWA:        "There  seems  to  be  a  general  consensus  that 
p.  32      favorable  attitudes  are  associated  with 
successful  implementation." 

JS :         HI  "...  significant  impact" 
p.  32       H2  " . . .  significant  impact" 

RR  Alt:     None 


Martin,  M.P.  and  J.E.  Trumbley,  "Measuring  Performance  of 
Automated  Systems,"  Journal    of   Systems   Management,     (February 
1986),,  pp.  7-17. 

"measuring  success  of  automated  systems  . . .  systematic 
structure  for  measuring" 

CWA:        No 

"Kucher' s  key  concept  here  is  that  of  comparison" 


NA  MC 


JS: 

P- 

14 

RR 

Alt 
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Barki,  H.,  and  S.L.  Huff,  "Change,  Attitude  to  Change  and 
Decision  Support  System  Success,"  Information    and 
Management,    Vol.  9,  (1995),  pp.  261-268. 

relationships  between  the  extent  of  changes  caused 
by  the  systems  to  users'  work  environments  and  the 
users'  attitudes  towards  work  related  changes 

concludes  DSS  users  are  more  satisfied  with  and  use 
more  extensively  those  DSS  that  bring  change  than 
those  that  do  not 

CWA:        "To  measure  system  success,  researchers  have 
p.  262      mainly  relied  on  either  of  two  proxies:  user 
satisfaction  or  system  use" 

JS :         "The  present  study  was  conducted  to  develop 
p.  262      empirical  evidence  concerning  ...  DSS  success" 

"User  satisfaction  is  a  more  appropriate  proxy 
[for  measuring  system  success]" 

RR  Alt:     Yes 

p.  262 


Hogue,  J.T.,  and  H.J.  Watson,  "An  Examination  of  Decision- 
Makers'  Utilization  of  Decision  Support  System  Output, " 
Information    &   Management,    Vol.  8,  No.  4,  (August  1985),  pp. 
205-212. 

Examines  DSS  contribution  to  decision  making  process 

Dss  users  are  highly  satisfied  with  their  systems 

CWA:        No 

JS:         "...  the  evaluation  of  user  satisfaction,  or  the 
p.  211      measurement  of  the  value  or  success  of  the  DSS" 

RR  Alt:     NA 


66 


Srinivasan,  A.,  "Alternative  Measures  of  System 
Effectiveness:  Associations  and  Implications,"  Management 
Information    Systems    Quarterly,    Vol.  9,  No.  3,  (September 
1985),  pp.  243-253. 

Reexamines  the  relationship  between  user  perceived 
effectiveness  measures  (user  satisfaction)  and 
behavioral  measures  (use) 

no  positive  association  found,  contrary  to  much  of 
existing  MIS  research 

CWA:        "While  acknowledging  the  importance  of  economic 
p.  243      analysis  of  MIS  value,  researchers  responded  to 
the  shifting  emphasis  from  efficiency  to  user 
effectiveness  by  focusing   either  on  MIS  usage  or 
user  perceived  effectiveness  [satisfaction].  Much 
of  the  MIS  literature  of  late  uses  one  of  the 
other . " 

JS :         "Measuring  and  Improving  Information  System 
p.  243      Effectiveness/Productivity  ...  Approaches  that 

have  been  advocated  include  . . .  user 

satisfaction" 

RR  Alt:     Yes 


Baily,  J.E.,  and  S.W.  Pearson,  "Development  of  A  Tool  for 
Measuring  and  Analyzing  Computer  User  Satisfaction, " 
Management    Science,    Vol.  29,  No.  5,  (May  1983),  pp.  530-545. 

tool  for  measuring  and  analyzing  computer  user 
satisfaction 

CWA:        "It  is  well  recognized  that  productivity  ... 

p.  530      Further  it  is  argued  that  utilization  is  directly 

connected  too  the  user  communities'  sense  of 

satisfaction  ..." 

JS :         "it  has  been  argued  that  user  satisfaction  is 
p.  530      correlated  to  information  system  utilization  and 
system  success" 

RR  Alt:     None 


67 


Ives,  B.,  M.H.  Olson,  and  J.J.  Baroudi,  "The  Measurement  of 
User  Information  Satisfaction,"  Comm    of  ACM,    Vol.  26,  No. 
10,  (October  1983),  pp.  785-793. 

Produces  a  short  form  measure  of  user  satisfaction 

CWA:        "Satisfaction  of  users  with  their  information 
p.  785      systems  is  a  potentially  measurable,  and 

generally  acceptable,  surrogate  for  utility  in 

decision  making." 

JS :         "MIS  provides  a  meaningful  Surrogate'  for  the 
p.  785      critical  but  unmeasurable  result  of  an 

information  system,  namely,  changes  in 

organizational  effectiveness." 

RR  Alt:     Yes 

p.  785 


King,  W..R.,  and  B.J.  Epstein,  "Assessing  Information  System 
Value:  An  Experimental  Study,"  Decision   Sciences,    Vol.  14, 
No.  1,  (January  1983),  pp.  34-45. 

manager  assessments  of  the  importance  of  a  Hubritis 
are  used  to  assess  information  system  value. 

CWA:        No 

JS:         "We  want  to  determine  whether  practicing 
p.  34       managers'  overall  evaluations  of  ^total  systems' 
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from  linear  models  that  incorporate  the  managers' 
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user  satisfaction  is  a  measure  for  DSS 

CWA:        "User  satisfaction  is  a  good  measure  of  system 
p.  171      effectiveness.  Furthermore,  it  is  argued  that  use 
of  a  system  is  directly  related  to  the  user 
communities'  sense  of  satisfaction.  Measuring  and 
analyzing  user  satisfaction  has  been  of  great 
interest  ever  since  the  dawn  of  the  computer 
era .  " 

JS :         "user  satisfaction  is  a  good  measure  of  system 
p.  171      effectiveness" 

RR  Alt:     None 
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Value  of  Information,"  Omega,    Vol.  10,  No.  3,  (1982),  pp. 
249-258. 

multidimensional  attribute  approach  to  assess  the 
value  of  information 

CWA:        No 

JS:         "predictions  of  total  system  evaluations  were 
p.  255      made  on  the  basis  of  self-explicated  attribute 
levels  and  weightings.  These  predictions  were 
then  compared  with  the  actual  overall  system 
assessments  that  were  made  by  the  respondents." 
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pp.  59-62. 

user  expectations  as  predictors  of  project  success 

or  failure 

CWA:        "The  method  for  measuring  . . .  success  . .  the 
p.  467      literature  has  suggested  both  attitudinal  (eg. 
user  satisfaction)  ..." 

JS :         "There  are  basis  in  the  literature  on  MIS 
p.  463      implementation  and  in  other  fields  to  support  the 
contention  that  expectations  formed  early  in  an 
MIS  development  project  should  be  related  to  the 
eventual  success  or  failure  of  that  project." 

RR  Alt:     None 


Hamilton,  S.,  and  N.L.  Chervany,  "Evaluating  Information 
System  Effectiveness  -  Part  I:  Comparing  Evaluation 
Approaches, "  Management    Information    Systems   Quarterly, 
(September  1981),  pp.  55-69. 

compares  evaluator  (user,  MIS  development,  internal 
audit,  and  top  management  personnel)  viewpoints  on 
system  effectiveness 

CWA:        "User  satisfaction  with  the  MIS  . . .  provides  the 
p.  79      most  useful  assessment  of  system  effectiveness. 
Substantial  evidence  supports  the  validity  ..." 

JS:        "the  use  of  a  standardized  validated  instrument 
p.  84       to  measure  system  success" 

RR  Alt:     None 


70 


Larcker,  D.F.,  and  P.V.  Lessig,  "Perceived  Usefulness  of 
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Instrument  for  measuring  perceived  usefulness 

CWA:        "...  the  ability  of  MIS  designers  to  specify 
p.  121      information  that  is  perceived  as  useful  by 

managers  is  instrumental  to  the  success  (or 

failure)  of  the  system" 

JS:         (see  CWA  above) 
p.  121 

RR  Alt:     None 


Deese,  D.R.,  "Experiences  Measuring  User  Satisfaction," 
Proceedings    of   the    Computer  Measurement    Group    of  ACM, 
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Employs  Pearson's  user  satisfaction  tool  to 
supplement  information  on  FEDSIM  review  of  data 
processing  installations 

CWA:        NA 

JS:         "to  supplement  information  ..." 
p.  59 

RR  Alt:     NA 


Neumann,  S.,  and  E.  Segev,  "A  Case  Study  of  User  Evaluation 
of  Information  Characteristics  for  Systems  Improvement," 
Information    &   Management,    Vol.  2,  (1979),  pp.  271-278. 

Questionnaire  used  to  evaluate  information  system 

No 


"In  addition  to  the  relative  contribution  of 
information  systems  ..." 

No 
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CWA: 

p.    35 

JS: 

P. 

35 

RR 

Alt: 

Robey,  D.,  "User  Attitudes  and  Management  Information  System 
Use,"  Academy   of  Management    Journal,    Vol.  22,  No.  3, 
(September  1979),  pp.  527-538. 

attitudes  are  positively  related  to  use 

CWA:         No 

JS :         "The  primary  objective  of  this  research  was  to 
p.  533      relate  the  objective  measures  of  system  use  to 

attitudes . " 
p.  537      "Strong  positive  relationships  have  been 

demonstrated  between  specific  user  attitudes  and 
actual  use  of  an  MIS.  It  does  not  necessarily 
follow,  however,  that  attitudes  cause  behavior." 

RR  Alt:     None 


King,  W.R.  and  J.I.  Rodriguez,  "Evaluating  Management 
Information  Systems,"  Management    Information    Systems 
Quarterly,    Vol.  2,  No.  3,  (September  1978),  pp.  43-51. 

multiple  criteria  framework  for  evaluating  MIS 

CWA:        No 

JS:        "These  form  broad  areas  of  value  assessments  - 
p.  45       attitudes,  value  perceptions,  information  usage 
and  decision  performance  -  may  be  measured  at 
various  stages  of  the  design  and  development 
process  to  evaluate  the  MIS." 

RR  Alt:     NA 
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Lucas,  H.C.,  Jr.,  "Application  and 

Implementation/Unsuccessful  Implementation:  The  Case  of  a 
Computer-Based  Order  Entry  System,"  Decision    Sciences,    Vol. 
9,  No.  1,   (January  1978),  pp.  68-79. 

"Successful  implementation  would  be  affected  by  the  quality 
of  service  and  the  personal  costs  and  benefits  to  the  users" 

CWA:        No 

"Satisfaction,  the  indicator  of  successful 
implementation . " 

Yes 


JS 

p. 

72 

RR 

Alt 

P. 
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Interfaces,    Vol.  8,  No.  3,  (February  1978),  pp.  70-75. 

analyze  the  adoption  and  rejection  of  the  same 
information  system  by  two  similar  departments  of  the 
same  company 

CWA:         No 

JS:        "Responses  show  the  user's  perception  of  how 

p.  71      favorable  the  system  is  for  each  area  of  concern 

Questionnaire  data  can  thus  provide  evidence  of 

specific  user  problems." 
p.  73       "while  we  cannot  say  that  attitudes  were  the 

cause  of  behavior..." 
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"The  Minnesota  Experiments  ...  examine  the  significance  of 
various  information  system  characteristics  on  decision 

activity . " 

CWA :         NA 

JS :         "In  addition  to  discussion  quality,  measures  of 
p.  917      user  evaluation  of  the  information  systems  used 
were  taken  ..." 

RR  Alt:      NA 
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"investigate  alternative  specifications  of  the  relationship 
between  user  influence  and  the  success  of  the  management 

information  systems" 

Outcome  variables:  Perceived  success  (i.e.,  user 
satisfaction) 

CWA:  "The  perceived  success  of  an  information 

system 

p.  590      is,  however,  not  the  only  possible  criterion  of 
success  and  it  seems  to  us  to  be  useful  to 
investigate  alternative  aspects  of  success." 

JS :         "the  measures  of  outcomes  [success]  are 
p.  589     perceptual  measures  ..." 

RR  Alt:     None 
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"Model  exploring  the  relationships  between  management 
information  system  users'  perceptions  of  their  computer 
system,  perceived  variables  exogenous  to  the  system, 
attitudes  and  system  usage." 

CWA:         No 

JS :         "According  to  the  model,  the  beliefs  about 
p.  579      characteristics  of  the  MIS  are  evaluated  and 
result  in  some  feeling  of  f avorableness  or 
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RR  Alt:     None 
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A  method  of  determining  the  monetary  value  of  an 
existing  MIS  report  is  based  on  user  perceptions  and 
the  semantic  differential  technique 

CWA:         No 

JS:        "it  seems  fair  to  conclude  that  the  EAB  system 
p.  55  2     was  well  thought  of  ...  as  a  whole" 

RR  Alt:     Yes 

pp.  4  6,  4  7 
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measurement  of  involvement  and  appreciation 
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p.  1180     will  increase  his  inquiry  involvement." 

RR  Alt:     No 
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Research  to  identify  "the  correlates  of  MIS  project 
success" 

CWA:         No 

JS :         "user  satisfaction  -  attitudes  of  the  managers 
p.  150      receiving  project  products  relative  to  how  well 
their  information  needs  were  being  satisfied." 

RR  Alt:     NA 
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APPENDIX  E.  TABLES  OF  FINDINGS 
TABLE  I .  STUDIES  MAKING  THE  CWA 


Author 

Year 

Barki 

90 

Conrath/Mignen 

9  0 

Hufnagel 

90 

Barooudi/Orlinkowski 

88 

Guimaraes /Gupta 

88 

Tati/Vessey 

8S 

Miller/Doyle 

87 

Baroudi /Olson/ Ives 

86 

Franz/Robey 

86 

Franz /Robey/Koebl it z 

86 

Barki/Huff 

85 

Srivinason 

85 

Bailey/Pearson 

83 

Ives /01s on /Baroudi 

83 

Mahmood 

83 

Ginzberg 

81 

Hamilton/Chervany 

81 

Larker/Vessig 

80 

TABLE  II.  BREAKDOWN  OF  THE  CWA 


All  Yrs 

18/40 

45% 

'83  + 

15/23 

65% 

'  77  + 

18/35 

51% 

77 


TABLE  III.  STUDIES  WITH 

JUSTIFICATION  SUPPORTED 

AUTHOR 

YEAR 

CWA 

Conrath/Mignen 

90 

Y 

Hufnagel 

90 

Y 

Evans,  et  al . 

88 

NA  (MC) 

Gallupe/De Sanctis 

88 

NA  (MC) 

Aldag/Power 

86 

NA  (MC) 

Baroudi/Olson/Ives 

86 

Y 

Franz/Robey 

86 

Y 

Franz /Robey/Koeblitz 

86 

Y 

Martin/Trumbley 

86 

NA  (MC) 

Srinivason 

85 

Y 

Bailey /Pear son 

83 

Y 

Ives /Olson /Bar oudi 

83 

Y 

King/Epstein 

83 

N 

Deese 

79 

NA  (FEDSIM) 

Nueman/Segev 

79 

N 

Robey 

79 

N 

King /Rodriguez 

778 

N 

Robey/Zellar 

78 

N 

Dickson/Senn/Chervany 

77 

NA  (MC) 

Schewe 

76 

N 

Gallagher 

74 

N 

Lucas 

74 

N 

Swanson 

74 

N 

Powers /Dickson 

73 

N 

TABLE  IV.  BREAKDOWN  OF  JUSTIFICATION  SUPPORTED 


All  Yrs. 

24/40 

60% 

'83  + 

13/23 

57% 

+CWA 

8/18 

44% 

'  83  + 

8/13 

62% 

'  77  + 

8/18 

44% 

78 


TABLE  V.  STUDIES  CITING  A  REFERENCE  REJECTING  ALTERNATIVES 

(RR  Alt) 


AUTHOR 

YEAR 

CWA 

Doll/Torkzadeh 

88 

N 

Barki/Huff 

85 

Y 

Srinivason 

85 

Y 

Ives /Olson /Bar oudi 

83 

Y 

Lucas 

78 

N 

Gallagher 

74 

N 

TABLE  VI.  BREAKDOWN  OF  RR  ALT 


All  Yrs 

6/32 

19% 

+CWA 

3/18 

17% 

'83  + 

3/13 

23% 

'  77  + 

3/18 

17% 

79 


TABLE  VII.  NAMES,  DEFINITION  AND  CONSTRUCT  OF  USER 
SATISFACTION  VARIABLES 


STUDY 
YEAR 

VARIABLE:  DEFINITION 
CONSTANT 

1.   Powers/Dickson 
'  73 

user  satisfaction:  attitudes  of  managers 
not  described 

2.   Swanson 
'74 

appreciation:  users  beliefs  about  relative  value 
not  described 

3.   Lucas 
'74 

user  attitudes:  not  defined 
some  scale  not  described 

4.   Gallagher 
'  74 

user  perceptions:  not  defined 
semantic  differential 

5 .   Schewe 
'76 

user  perceptions,  attitudes:  not  defined 
not  described 

6.   Edstrom 

'  77 

perceived  success:  not  defined 
complete  success  -  complete  failure 

7.   Dukson/Senn/Chervany 
'  77 

user  evaluation:  not  defined 
not  described 

8.   Robey/Zeller 
'  78 

attitudes:  not  defined 
Shultz-Slevin 

9.   Lucas 
'  78 

user  perceptions:  modified  JDI 
Shultz-Slevin 

10.  King/Rodriguez 
'  78 

value  perceptions:  "direct  assessment"  of  MIS 
Shultz-Slevin 

11.  Robey 
'  79 

user  attitudes:  not  defined 
Shultz-Slevin 

12.  Neumann/Segev 
'79 

user  evaluation:  accuracy,  content,  frequency,  recency 

questionnaire 

1  3  .  Deese 
'  79 

user  satisfaction:  * 
Pearson  method 

14.  Larkin/Lessig 
'80 

perceived  useableness/perceived  importance 

questionnaire  -  7  pt  scale 

15.  Hamilton/Chervany 
'81 

user  satisfaction:  * 
Pearson  method 

16.  Ginzberg 
'81 

user  satisfaction 
Shultz-Slevin 

17.  Epstein/King 
'83 

satisfaction/value  perception:  10  attributes 
guest ionnarie,  5  pt  scale 

18.  Mahmood 
'83 

user  satisfaction:  * 
Pearson  method 

19.  King/Epstein 
'83 

satisfaction/value  perception:  10  attributes 
questionnaire,  5  pt  scale 

20.  Ives/Olson/Baroudi 
'83 

user  satisfaction:  * 
Pearson,  Bailey/Pearson 

21.  Bailey/Pearson 
'83 

user  satisfaction:  semantic  differential  factors  39 
Pearson  method 

22.  Srinivasen 
'85 

user  satisfaction:  * 
Bailey /Pear son 

23.  Hogue/Watson 
'85 

perceived  value:  satisf action/diissatisfact ion 
questionnaire  7  pt  scale 

80 


STUDY 
YEAR 


24.  Bark. 
'85 


25.  Martin/Trumbly 
'86 


26.  Frar.z/Robey/Koebiitz 
'86 


27.  Franz/Robey 
'86 


28.  Baroudi/Olson/Ives 
'86 


29.  Aldag/Power 
'86 


30.  Miller/Doyle 
|_87 

31.  Tart/Vessey 


32.  Gurmaraes/Gupta 


33.  Gallupe/DeSanctis 


34.  Evans,  et.  al 


35.  Doll/Torkzadeh 


36.  Baroudi/Orlikowski 


37.  Hufnagel 
[_90 

38.  Conrath/Mignen 

'90 


39.  Burkhard 
'  90 


40.  Barki 
'90 


VARIABLE:  DEFINITION 
CONSTANT 


user  satisfaction: 
Pearson  Method 


user  accommodation:  "r^le  of  satisficing' 

questionnaire  -  Likert  scales 


user  attitudes/perceptions 
Schuitz-Sievin 


perceivea  usefulness:  user  satisfaction 
Bailey  and  Pearson 


user  satisfaction:  * 
Bailey  and  Pearson 


user  perceptions:  multiple  item 
Likert  scale 


user  satisfaction:  * 
Bailey  and  Pearson 


user  satisfaction: 
Bailey  and  Pearson 


user  satisfaction:  * 
Bailey  and  Pearson 


perceived  reactions,  satisfaction 
questionnaire,  Likert  scale 


user  satisfaction:  * 
Pearson  Method 


user  satisfaction:  * 
Bailey  and  Pearson 


user  satisfaction:  short  form  13  factors 
Bailey  and  Pearson 


user  satisfaction:  * 
Bailey  and  Pearson 


user  satisfaction: 
Bailey  and  Pearson 


user  satisfaction:  * 

Bailey  and  Pearson  short  forri 


user  satisfaction:  * 
Bailey  and  Pearson 


see  "USER  SATISFACTION"  in  Section  III,  METHODOLOGY 
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TABLE  VIII.  STUDIES  WITH  VALIDITY/RELIABILITY  CHECKS 


Hufnagel 

'90 

Baroudi/Orlikowski 

'88 

Doll/Torkzadeh 

'88 

Gurmaraes /Gupta 

'88 

Tart/Vessey 

'88 

Miller/Doyle 

'87 

Bailey/Pearson           '83 

Ives/Olson/Baroudi 

'83 

Mahmood 

'83 

Larcker/Lessey 

'80 
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